Compare commits

...
100 Commits
Author SHA1 Message Date
dpanigra 818175a18d bug: remove A100 as recommended specs 2026-02-25 14:00:42 +00:00
Damodar PanigrahiandGitHub b468809df7 bug: ahref update (#4449)
* bug: ahref update

* fix: linter

* fix: typo fix
2026-02-25 13:46:19 +00:00
Sam-DecigaandGitHub 0417d8b9c4 feat: Deprecate Claude 3 Haiku (#4448)
Deprecation start date: Feb. 23, 2026
End of Support date: Aug. 23, 2026
b/485993204
2026-02-23 20:57:43 -05:00
Damodar PanigrahiandGitHub 7750e83fbb fix: inference key change, finetuning jaxlib update (#4447) 2026-02-23 19:45:59 +00:00
Damodar PanigrahiandGitHub 649800e646 feat: Alphagenome finetuning notebook (#4445)
* feat: Alphagenome finetuning notebook

* Update cloudai_alphagenome_finetune.ipynb

Fixed the lint errors

* Update cloudai_alphagenome_finetune.ipynb

Fix lint errors

* feat: Add Alphagenome finetune

* feat: Include the Alphagenome finetuning notebook url in the readme. Update the codeowners

* feat: Add alphagegenome finetune notebook to the readme, add user to codeowners

* feat: fix spelling
2026-02-20 14:13:36 +00:00
Mend RenovateandGitHub 42b35056fa chore(deps): update dependency black to v26 (#4422) 2026-02-18 15:07:12 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
24974eda95 Bump protobuf (#4437)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 4.25.8 to 5.29.6.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Commits](https://github.com/protocolbuffers/protobuf/commits)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-version: 5.29.6
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-18 15:06:37 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ad41377783 Bump protobuf (#4438)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 4.25.8 to 5.29.6.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Commits](https://github.com/protocolbuffers/protobuf/commits)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-version: 5.29.6
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-18 15:06:10 +00:00
Sam-DecigaandGitHub 36dea3ca01 feat: Anthropic Sonnet-4-6 launch (#4442)
Signed-off-by: Sam-Deciga <decigagarcia@google.com>
2026-02-17 14:02:30 -05:00
b75b2ea4d7 fix: updated with latest whl file version : alphagenome-0.4.2.6-py3-none-any.whl (#4439)
Co-authored-by: hyper-param <peeyusht@google.com>
2026-02-10 18:27:03 +00:00
Vertex MG TeamandCopybara-Service 28f7fc4445 Add SAM 3 notebook to Vertex AI Model Garden.
PiperOrigin-RevId: 866598579
2026-02-06 13:45:47 -08:00
Rayan DasoriyaandCopybara-Service bf2c1226fd Update the license year
PiperOrigin-RevId: 866214899
2026-02-05 19:07:14 -08:00
Vertex MG TeamandCopybara-Service 2990c53292 Added notebook sample for batch inference using the remote sensing VMG models
PiperOrigin-RevId: 866061602
2026-02-05 12:27:10 -08:00
Sam-DecigaandGitHub 531d9cfee0 feat: New Anthropic model (#4436) 2026-02-05 14:07:05 -05:00
Vertex MG TeamandCopybara-Service ff18ec7af5 Add --total-gpus to multi-model model-cohost deployment config.
PiperOrigin-RevId: 863070590
2026-01-29 22:39:06 -08:00
Sam-DecigaandGitHub 008eb409ef feat: NVIDIA-Llama-Nemotron-Super-49B (#4430) 2026-01-29 08:51:52 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b2dba4b568 Bump pyasn1 (#4421)
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.1 to 0.6.2.
- [Release notes](https://github.com/pyasn1/pyasn1/releases)
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst)
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.1...v0.6.2)

---
updated-dependencies:
- dependency-name: pyasn1
  dependency-version: 0.6.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-27 20:14:34 +00:00
Sam-DecigaandGitHub 665547f790 feat:New MARS8 model (#4429)
* feat:New MARS8 model

* feat:New MARS8 model
2026-01-27 10:43:43 -05:00
Vertex MG TeamandCopybara-Service 5078c44eb8 Updated the bucket path for the remote sensing models.
PiperOrigin-RevId: 861234161
2026-01-26 09:47:01 -08:00
Bhaskar GoyalandGitHub da6e46531e feature: Retire Mistral 24.11 and Codestral 25.01 from Mistral Intro files. (#4427) 2026-01-23 18:47:36 +00:00
0a4091a3b1 fix: Fix json response parsing error (#4426)
Co-authored-by: hyper-param <peeyusht@google.com>
2026-01-23 13:58:03 +00:00
Sam-DecigaandGitHub 5afa83dd25 feat: MongoDB voyage-4 launch (#4419)
* feat: MongoDB voyage-4 launch

* feat: MongoDB voyage-4 launch

* feat: MongoDB voyage-4 launch

* feat: MongoDB voyage-4 launch
2026-01-16 16:26:05 -05:00
Sam-DecigaandGitHub 8cab85d6ad feat: MongoDB voyage-multimodal-3.5 launch (#4420)
* feat: MongoDB voyage-multimodal-3.5 launch

* feat: MongoDB voyage-multimodal-3.5 launch
2026-01-16 14:57:11 -05:00
Eric DongandGitHub a7f3940635 refactor: Update Github icon (#4418) 2026-01-13 13:36:20 -05:00
Mend RenovateandGitHub 5bb1a75a48 chore(deps): update dependency pyupgrade to v3.21.2 (#4359) 2026-01-13 18:26:35 +00:00
Damodar PanigrahiandGitHub 8fe4985aa8 feat: WeatherNext2 Initial Updates (#4417) 2026-01-13 13:24:48 -05:00
intentsolutions.ioandGitHub 996b6534d9 Add ADK inline source deployment tutorial for Agent Engine (#4393)
* Add ADK inline source deployment tutorial for Agent Engine

* fix: address Gemini review feedback

- Change model from gemini-2.0-flash to gemini-1.5-flash-001
- Improve exception handling with ZoneInfoNotFoundError

* fix: address Gemini code review feedback

- Use specific ZoneInfoNotFoundError exception instead of generic Exception
- Define REQUIREMENTS variable once and reuse to avoid duplication
- Keep generic Exception as fallback for unexpected errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* style: fix notebook formatting via official linter
2026-01-13 08:56:07 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1061ae5348 Bump urllib3 (#4416)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.6.0 to 2.6.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.3)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-12 20:01:22 +00:00
Eric DongandGitHub 4fac3a630f refactor: reformat notebook template (#4414)
* refactor: reformat notebook template

* Update Python version to 3.12

* Fix an import

* Update licience year
2026-01-12 13:15:30 -05:00
Eric DongandGitHub 1f2db2903c refactor: Updated Github icons (#4413) 2026-01-12 10:10:34 -05:00
Sam-DecigaandGitHub 912a52de70 feat:MongoDB Voyage 3.5-Lite (#4403)
* feat:MongoDB Voyage 3.5-Lite

* chore:apply linter

* Update voyage-3.5-lite.ipynb

Fixing MODEL_NAME
2026-01-06 21:19:17 -05:00
Sam-DecigaandGitHub 5e29090e86 Model Deprecation (#4409)
Haiku 3.5 Model Deprecation
2026-01-05 16:30:23 -05:00
Mend RenovateandGitHub cd8fcd1839 Update actions/checkout action to v6 (#4374) 2026-01-05 14:20:36 +00:00
Mend RenovateandGitHub e51075ec4b chore(deps): update dependency black to v25.12.0 (#4360) 2026-01-05 14:18:08 +00:00
Mend RenovateandGitHub 9709c0dddb chore(deps): update dependency isort to v7 (#4289) 2026-01-05 14:14:26 +00:00
Mend RenovateandGitHub a21ae41762 Update dependency python to 3.14 (#4398) 2026-01-05 14:13:11 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ff5ff7b609 Bump urllib3 (#4386)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:12:43 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
23748f443e Bump urllib3 (#4387)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:12:18 +00:00
Vertex MG TeamandCopybara-Service fc6b2167de ComfyUI tutorial notebook
PiperOrigin-RevId: 851380881
2026-01-02 10:20:16 -08:00
e79a45358c Migrate gsutil usage to gcloud storage (#4299)
* Migrate gsutil usage to gcloud storage

* changes for 4299

* Apply automated linter fixes

* update

* remove model_garden changes

* revert to main

* revert model garden file

* Update model_garden_weather_prediction_on_vertex.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:37:14 -05:00
20d19fb11c Migrate gsutil usage to gcloud storage (#4331)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Changes for 4331

* Changes for 4331

* Removed changes model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:36:46 -05:00
9c9f7a6e2a Migrate gsutil usage to gcloud storage (#4334)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* changes for 4334

* fix linter issue\ for 4334

* manual changes

* Restore gcloud migration code

* Remove changes for model garden

* update

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:36:21 -05:00
ede41c2115 Migrate gsutil usage to gcloud storage (#4335)
* Migrate gsutil usage to gcloud storage

* remoed changes for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:35:53 -05:00
ca53786c04 Migrate gsutil usage to gcloud storage (#4322)
* Migrate gsutil usage to gcloud storage

* Changes for 4322

* fix linter issue for 4322

* removed changes for model garden

* Update model_garden_pytorch_gemma_peft_finetuning_hf.ipynb

* Update model_garden_pytorch_gemma_peft_finetuning_hf.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:49:46 -05:00
419f8310c9 Migrate gsutil usage to gcloud storage (#4316)
* Migrate gsutil usage to gcloud storage

* Changes for 4316

* Changes for 4316

* fix linter issue for 4316

* removed changes for model garden

* removed changes for model garden

* Update model_garden_tfvision_image_classification.ipynb

* Update model_garden_tfvision_image_classification.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:49:33 -05:00
996b690e03 Migrate gsutil usage to gcloud storage (#4317)
* Migrate gsutil usage to gcloud storage

* Changes for 4317

* Changes for 4317

* fix linter issue for 4317

* removed changes for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 15:48:30 +00:00
5b9d04d63a Migrate gsutil usage to gcloud storage (#4318)
* Migrate gsutil usage to gcloud storage

* Changes for 4318

* fix linter issue for 4318

* removed changes for model garden:

* Update model_garden_gemma2_deployment_on_vertex.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:47:49 -05:00
9ed3c2f83d Migrate gsutil usage to gcloud storage (#4320)
* Migrate gsutil usage to gcloud storage

* Changes for 4320

* fix linter issue for 4320

* removed chnages for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:47:17 -05:00
5fc93c8bdb Migrate gsutil usage to gcloud storage (#4328)
* Migrate gsutil usage to gcloud storage

* PR changes for 4328

* Fix Linter issue for 4328

* changes removed model garden

* Update model_garden_jax_fvlm.ipynb

* Update model_garden_jax_fvlm.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:46:52 -05:00
3cf46226e9 Migrate gsutil usage to gcloud storage (#4321)
* Migrate gsutil usage to gcloud storage

* Changes for 4321

* fix the linter issue for br 4321

* removed changes for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:46:04 -05:00
439f6a0cae Migrate gsutil usage to gcloud storage (#4329)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Manual Changes

* Linter fiex the issues for 4329

* Revert "Linter fiex the issues for 4329"

This reverts commit a1ed9c6f59.

* Revert "Manual Changes"

This reverts commit 8ca9b56c5b.

* changes removed model garden

* Update model_garden_gemma2_finetuning_on_vertex.ipynb

* Update model_garden_pytorch_llama3_3_finetuning.ipynb

* Update model_garden_pytorch_llama3_3_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:45:30 -05:00
993898bb71 Migrate gsutil usage to gcloud storage (#4330)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Removed blank line and spaces Manual Changes

* fix the Linter issue for 4330

* Changes for model garden

* Update model_garden_pytorch_llama3_1_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 15:45:00 +00:00
9e9e639375 Migrate gsutil usage to gcloud storage (#4323)
* Migrate gsutil usage to gcloud storage

* Manual Changes-Migrate gsutil usage to gcloud storage

* Manual Changes

* Fix linter issue for 4323

* removed changes for model garden

* Update model_garden_axolotl_gpt_oss_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:44:34 -05:00
633cf6a799 Migrate gsutil usage to gcloud storage (#4327)
* Migrate gsutil usage to gcloud storage

* Manual Changes-Migrate gsutil usage to gcloud storage

* Changes for 4327

* fixing linting erro

* removed changes of model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:44:05 -05:00
8b618bc455 Migrate gsutil usage to gcloud storage (#4326)
* Migrate gsutil usage to gcloud storage

* Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'

* Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'

* Revert "Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'"

This reverts commit 175eaa4fe8.

* Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'

* Changes for 4326

* Changes for 4326

* Linter fix issue for 4326

* removed model garden changes

* Update model_garden_movinet_action_recognition.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:43:31 -05:00
6bbe3bcfe0 Migrate gsutil usage to gcloud storage (#4336)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* removed model garden changes

* Update model_garden_llama3_1_finetuning_with_workbench.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:42:44 -05:00
Margubur RahmanandGitHub 814827ac19 Migrate gsutil usage to gcloud storage (#4332) 2025-12-25 10:42:12 -05:00
Margubur RahmanandGitHub 7a613785b9 Migrate gsutil usage to gcloud storage (#4333) 2025-12-25 15:41:38 +00:00
100243e90a Migrate gsutil usage to gcloud storage (#4338)
* Migrate gsutil usage to gcloud storage

* Manual Changes

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
2025-12-25 15:40:46 +00:00
6424515b03 Migrate gsutil usage to gcloud storage (#4339)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Manual Changes

* Fix: Updated gcloud storage command without formatting

* Manual Changes

* Manual Changes

* Revert "Manual Changes"

This reverts commit a7a7bda0f9.

* Manual Changes

* Revert "Manual Changes"

This reverts commit a7a7bda0f9.

* Manual Changes

* Revert "Manual Changes"

This reverts commit 71c777d5f1.

* Manual Changes

* Manual changes

* Changes for 4339

* Changes for 4339

* Changes for 4339

* Fix: Applied linter formatting and resolved style issues

* gcloud to gsutilchanges for  4339

* removed gsutil to gcloud migration

* Manual changes

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 15:40:00 +00:00
8d22b221b4 Migrate gsutil usage to gcloud storage (#4337)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Manual Changes

* Manual Changes

* Manual Changes

* Revert "Manual Changes"

This reverts commit 3ef23f1456.

* Manaul Changes

* Changes for 4337

* Fix: Resolved linter errors and formatted notebooks

* Revert "Fix: Resolved linter errors and formatted notebooks"

This reverts commit 7534760fe1.

* Revert "Changes for 4337"

This reverts commit f497568138.

* Changes for 4337

* gcloud to gsutil migration

* removed changes for model garden

* Update model_garden_axolotl_qwen3_finetuning.ipynb

* Update model_garden_axolotl_qwen3_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:39:31 -05:00
2a8ad7cdbb Migrate gsutil usage to gcloud storage (#4319)
* Migrate gsutil usage to gcloud storage

* changes for 4319

* changes for 4319

* Apply automated linter fixes

* remove unused import

* removed model_garden changes

* Update model_garden_pytorch_mixtral_peft_tuning.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:41:58 -05:00
3239b301f2 Migrate gsutil usage to gcloud storage (#4315)
* Migrate gsutil usage to gcloud storage

* changes for 4315

* Apply automated linter fixes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:41:28 -05:00
acb10d14b8 Migrate gsutil usage to gcloud storage (#4314)
* Migrate gsutil usage to gcloud storage

* changes for 4314

* Apply automated linter fixes

* remove unused imports

* removed model_garden changes

* updates

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:41:00 -05:00
778d145970 Migrate gsutil usage to gcloud storage (#4313)
* Migrate gsutil usage to gcloud storage

* changes for 4313

* Apply automated linter fixes

* remove model_garden updates

* remove model_garden updates

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:40:34 -05:00
4d00356f4b Migrate gsutil usage to gcloud storage (#4312)
* Migrate gsutil usage to gcloud storage

* changes for 4312

* Apply automated linter fixes

* remove model_garden folder changes

* remove model_garden folder changes

* Update model_garden_pytorch_llama2_peft_hyperparameter_tuning.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:40:01 -05:00
0654305994 Migrate gsutil usage to gcloud storage (#4311)
* Migrate gsutil usage to gcloud storage

* changes for 4311

* Apply automated linter fixes

* removed unused imports

* remove model_garden changes

* Update model_garden_pytorch_zipnerf.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:39:27 -05:00
c53f392c5a Migrate gsutil usage to gcloud storage (#4310)
* Migrate gsutil usage to gcloud storage

* changes for 4310

* Apply automated linter fixes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:38:53 -05:00
f4b56e92ae Migrate gsutil usage to gcloud storage (#4308)
* Migrate gsutil usage to gcloud storage

* changes for 4308

* added = in command

* Apply automated linter fixes

* removed model_garden folder changes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:38:24 -05:00
Margubur RahmanGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gurusai-voletigemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
45ec1cf18a Migrate gsutil usage to gcloud storage (#4307)
* Migrate gsutil usage to gcloud storage

* changes for 4307

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Apply automated linter fixes

* Update sdk_vector_search_for_indexing.ipynb

* Update sdk_vector_search_for_indexing.ipynb

* update

* Update model_garden_pytorch_falcon_instruct_quantization.ipynb

* Update model_garden_pipeline_templates_t5x.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-24 14:37:45 +00:00
f6c8bcf937 Migrate gsutil usage to gcloud storage (#4301)
* Migrate gsutil usage to gcloud storage

* changes for 4301

* linter changes

* Revert "linter changes"

This reverts commit 6665133b2c.

* Apply automated linter fixes

* Update training-multi-class-classification-model-for-ads-targeting-usecase.ipynb

* Update training-multi-class-classification-model-for-ads-targeting-usecase.ipynb

* removed model_garden folder changes

* Update model_garden_mediapipe_object_detection.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:37:11 -05:00
Rayan DasoriyaandCopybara-Service 9e590d5a9f Update notebooks based on latest deployment options
PiperOrigin-RevId: 847555297
2025-12-21 19:32:54 -08:00
46e0ea4f1c Migrate gsutil usage to gcloud storage (#4309)
* Migrate gsutil usage to gcloud storage

* changes for 4309

* changes for 4309

* Apply automated linter fixes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-20 11:34:42 -05:00
300fce6b9f Migrate gsutil usage to gcloud storage (#4305)
* Migrate gsutil usage to gcloud storage

* Apply automated linter fixes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-20 11:34:10 -05:00
Vertex MG TeamandCopybara-Service 52e3066c38 No public description
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 845978812
2025-12-19 10:42:55 -08:00
27ebf52198 Migrate gsutil usage to gcloud storage (#4296)
* Migrate gsutil usage to gcloud storage

* updates

* update

* Update llm_streaming_prediction.ipynb

* revert change

* removed model_garden folder changes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 11:57:20 -05:00
ca7d4e153e Migrate gsutil usage to gcloud storage (#4297)
* Migrate gsutil usage to gcloud storage

* changes for 4297

* Apply automated linter fixes

* removed changes from model_garden

* removed changes from model_garden

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 11:56:49 -05:00
5b6c766629 Migrate gsutil usage to gcloud storage (#4298)
* Migrate gsutil usage to gcloud storage

* update

* linter changes for 4298

* Revert "linter changes for 4298"

This reverts commit c7a00a9710.

* Linter fixes

* update

* update

* Update distributed_hyperparameter_tuning.ipynb

* removed changes in model_garden folder

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 11:56:15 -05:00
Margubur RahmanandGitHub 5efa51206f Migrate gsutil usage to gcloud storage (#4324) 2025-12-19 15:11:28 +00:00
Margubur RahmanandGitHub e604a4d43e Migrate gsutil usage to gcloud storage (#4341) 2025-12-19 15:10:52 +00:00
23af5373ec Migrate gsutil usage to gcloud storage (#4295)
* Migrate gsutil usage to gcloud storage

* removed notes

* Apply automated linter fixes

* Revert "Apply automated linter fixes"

This reverts commit 3f6c4c0d36.

* linter changes

* update

* update

* update

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 10:09:05 -05:00
7577c0b1fc Migrate gsutil usage to gcloud storage (#4292)
* Migrate gsutil usage to gcloud storage

* Manual change

* update

* Update NotebookProcessors.py

* Update NotebookProcessors.py

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 15:07:50 +00:00
b648f9e73b Migrate gsutil usage to gcloud storage (#4306)
* Migrate gsutil usage to gcloud storage

* changes for 4306

* Apply automated linter fixes

* revert gsutil to gcloud

* Update model_garden_pytorch_stable_diffusion_custom.ipynb

* Update model_garden_pytorch_stable_diffusion_xl_lcm.ipynb

* Update model_garden_pytorch_deployed_model_agent_engine.ipynb

* Update model_garden_pytorch_blip_vqa.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 15:07:21 +00:00
Margubur RahmanandGitHub 966bbc49a7 Migrate gsutil usage to gcloud storage (#4340) 2025-12-18 13:50:35 -05:00
f5d341ae45 Migrate gsutil usage to gcloud storage (#4303)
* Migrate gsutil usage to gcloud storage

* changes for 4303

* Apply automated linter fixes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:49:38 -05:00
70770a50c7 Migrate gsutil usage to gcloud storage (#4302)
* Migrate gsutil usage to gcloud storage

* changes for 4302

* changes for 4302

* removed note

* linter changes

* Revert "linter changes"

This reverts commit a9544e8251.

* Apply automated linter fixes

* Update lightweight_functions_component_io_kfp.ipynb

* Update lightweight_functions_component_io_kfp.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:48:39 -05:00
Matej AleksandrovandCopybara-Service f181c39cbf No public description
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 845941350
2025-12-18 10:04:35 -08:00
Rayan DasoriyaandCopybara-Service a5637f87f2 Add notebook for T5Gemma 2 local inference
PiperOrigin-RevId: 846315479
2025-12-18 10:03:26 -08:00
0103299084 Migrate gsutil usage to gcloud storage (#4304)
* Migrate gsutil usage to gcloud storage

* changes for 4304

* Apply automated linter fixes

* remove unused import

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 14:00:23 +00:00
1867536d76 Migrate gsutil usage to gcloud storage (#4294)
* Migrate gsutil usage to gcloud storage

* updates

* removed notes added by agent

* removed note

* linter changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:59:10 +00:00
0edae683e7 Migrate gsutil usage to gcloud storage (#4293)
* Migrate gsutil usage to gcloud storage

* update

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:58:03 +00:00
Matej AleksandrovandCopybara-Service 8471b5cb6f No public description
PiperOrigin-RevId: 845941350
2025-12-17 15:25:49 -08:00
Aaron DietzandGitHub 87f540ac53 Update notebook_template_review.py (#4405)
Minor change to help us update references to "custom training" to specify "serverless training"
2025-12-17 22:08:00 +00:00
Sam-DecigaandGitHub babeba9f02 feat:NVIDIA Nemotron Nano v2 12B VL - 2025-12-TBD (#4391)
* feat:NVIDIA Nemotron Nano v2 12B VL - 2025-12-TBD

* refactor:Reformat Notebook

* refactor:Reformat Notebook
2025-12-17 18:49:58 +00:00
Vertex MG TeamandCopybara-Service 85c649dd26 Add Llama 3.3 TPU7x deployment notebook.
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 845481841
2025-12-16 16:50:45 -08:00
Vertex MG TeamandCopybara-Service b7135ae1f0 Add Llama 3.3 TPU7x deployment notebook.
PiperOrigin-RevId: 845481841
2025-12-16 16:38:07 -08:00
Vertex MG TeamandCopybara-Service 2f5119a266 Allows the user to select spot VM for deployment
PiperOrigin-RevId: 845064375
2025-12-15 21:23:11 -08:00
Vertex MG TeamandCopybara-Service 23e64ca76f fix: Update TimesFM 2.0 deployment notebook to use GCS path as MODEL_ID
PiperOrigin-RevId: 844826585
2025-12-15 10:25:10 -08:00
gurusai-voletiandGitHub 0ba5a62cc9 Fix ci workflow to use python 3.13 to avoid linter issues (#4397)
* update

* use python 3.13
2025-12-15 13:54:49 +00:00
Damodar PanigrahiandGitHub 0be2c6fd0c feat: authenticate using sa (#4394) 2025-12-12 15:38:25 -05:00
629 changed files with 18092 additions and 8335 deletions
@@ -365,7 +365,7 @@ def process_and_execute_notebook(
# Use gcloud to get tail
try:
result.error_message = subprocess.check_output(
["gsutil", "cat", "-r", "-1000", log_file_uri], encoding="UTF-8"
["gcloud", "storage", "cat", "--range", "-1000", log_file_uri], encoding="UTF-8"
)
except Exception as error:
result.error_message = str(error)
+2 -2
View File
@@ -56,8 +56,8 @@ def execute_notebook(
print("\n=== DOWNLOAD EXECUTED NOTEBOOK ===\n")
print(f"Please debug the executed notebook by downloading the executed notebook:")
print("Option 1. Using gsutil. Run the following command in your terminal.")
print(f'\tgsutil cp "{output_file_or_uri}" .')
print("Option 1. Using gcloud storage. Run the following command in your terminal.")
print(f'\tgcloud storage cp "{output_file_or_uri}" .')
print("Option 2. Using this link.")
print(f"\thttps://storage.googleapis.com/{output_file_or_uri[5:]}")
+1 -1
View File
@@ -108,7 +108,7 @@ class VertexAIInstallProprocessor(Preprocessor):
if "google-cloud-aiplatform" not in content:
return content
return (
f"gsutil cp {self.vertex_ai_wheel} google-cloud-aiplatform.whl\n" +
f"gcloud storage cp {self.vertex_ai_wheel} google-cloud-aiplatform.whl\n" +
content.replace("google-cloud-aiplatform\n", "google-cloud-aiplatform.whl\n")
.replace("google-cloud-aiplatform ", "google-cloud-aiplatform.whl ")
)
+2 -2
View File
@@ -15,7 +15,7 @@ def download_file(bucket_name: str, blob_name: str, destination_file: str) -> st
remote_file_path = "".join(["gs://", "/".join([bucket_name, blob_name])])
subprocess.check_output(
["gsutil", "cp", remote_file_path, destination_file], encoding="UTF-8"
["gcloud", "storage", "cp", remote_file_path, destination_file], encoding="UTF-8"
)
return destination_file
@@ -27,7 +27,7 @@ def upload_file(
) -> str:
"""Copies a local file to a GCS path"""
subprocess.check_output(
["gsutil", "cp", local_file_path, remote_file_path], encoding="UTF-8"
["gcloud", "storage", "cp", local_file_path, remote_file_path], encoding="UTF-8"
)
return remote_file_path
+2 -2
View File
@@ -9,9 +9,9 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.x'
python-version: '3.12'
- name: Fetch pull request branch
uses: actions/checkout@v4
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Fetch base main branch
+3 -3
View File
@@ -2,9 +2,9 @@ git+https://github.com/tensorflow/docs
ipython
jupyter
nbconvert
black==25.1.0
pyupgrade==3.21.0
isort==6.0.1
black==26.1.0
pyupgrade==3.21.2
isort==7.0.0
flake8==7.3.0
nbqa==1.9.1
@@ -148,7 +148,7 @@ implementation:
# Downloading the model archive from GCS
# TODO: Fix gsutil bugs (requires project ID, has auth issues) and use gsutil instead.
# gsutil cp "$model_archive_uri" "$model_archive_local_path"
# gcloud storage cp "$model_archive_uri" "$model_archive_local_path"
pip install google-cloud-storage
python -c '
import sys
@@ -24,12 +24,12 @@ implementation:
# Checking whether the URI points to a single blob, a directory or a URI pattern
# URI points to a blob when that URI does not end with slash and listing that URI only yields the same URI
if [[ "$uri" != */ ]] && (gsutil ls "$uri" | grep --fixed-strings --line-regexp "$uri"); then
if [[ "$uri" != */ ]] && (gcloud storage ls "$uri" | grep --fixed-strings --line-regexp "$uri"); then
mkdir -p "$(dirname "$output_path")"
gsutil -m cp -r "$uri" "$output_path"
gcloud storage cp --recursive "$uri" "$output_path"
else
mkdir -p "$output_path" # When source path is a directory, gsutil requires the destination to also be a directory
gsutil -m rsync -r "$uri" "$output_path" # gsutil cp has different path handling than Linux cp. It always puts the source directory (name) inside the destination directory. gsutil rsync does not have that problem.
gcloud storage rsync --recursive "$uri" "$output_path" # gsutil cp has different path handling than Linux cp. It always puts the source directory (name) inside the destination directory. gsutil rsync does not have that problem.
fi
- inputValue: GCS path
- outputPath: Data
@@ -110,7 +110,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -192,7 +192,7 @@
},
"outputs": [],
"source": [
"! gsutil cp -r $gcs_output_uri_prefix/model ./model_server/"
"! gcloud storage cp --recursive $gcs_output_uri_prefix/model ./model_server/"
]
},
{
@@ -556,7 +556,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
},
{
@@ -412,7 +412,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
}
],
@@ -77,4 +77,4 @@ echo "After the job is completed successfully, model files will be saved at $JOB
# # Verify the model was exported
# echo "Verify the model was exported:"
# gsutil ls ${JOB_DIR}/
# gcloud storage ls ${JOB_DIR}/
@@ -34,4 +34,4 @@ RUN echo "service_envelope=json\n" "inference_address=http://0.0.0.0:${AIP_H
USER model-server
# run Torchserve HTTP serve to respond to prediction requests
CMD ["echo", "AIP_STORAGE_URI=${AIP_STORAGE_URI}", ";", "gsutil", "cp", "-r", "${AIP_STORAGE_URI}/${MODEL_NAME}.mar", "/home/model-server/model-store/", ";", "ls", "-ltr", "/home/model-server/model-store/", ";", "torchserve", "--start", "--ts-config=/home/model-server/config.properties", "--models", "${MODEL_NAME}=${MODEL_NAME}.mar", "--model-store", "/home/model-server/model-store"]
CMD ["echo", "AIP_STORAGE_URI=${AIP_STORAGE_URI}", ";", "gcloud", "storage", "cp", "--recursive", "${AIP_STORAGE_URI}/${MODEL_NAME}.mar", "/home/model-server/model-store/", ";", "ls", "-ltr", "/home/model-server/model-store/", ";", "torchserve", "--start", "--ts-config=/home/model-server/config.properties", "--models", "${MODEL_NAME}=${MODEL_NAME}.mar", "--model-store", "/home/model-server/model-store"]
@@ -67,4 +67,4 @@ echo "After the job is completed successfully, model files will be saved at $JOB
# # Verify the model was exported
# echo "Verify the model was exported:"
# gsutil ls ${JOB_DIR}/
# gcloud storage ls ${JOB_DIR}/
@@ -478,8 +478,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -498,8 +497,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -582,8 +580,7 @@
"outputs": [],
"source": [
"# Download the sample data into your RAW_DATA_PATH\n",
"! gsutil cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $RAW_DATA_PATH"
]
"! gcloud storage cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $RAW_DATA_PATH" ]
},
{
"cell_type": "code",
@@ -1621,9 +1618,7 @@
"! gcloud scheduler jobs delete $SIMULATOR_SCHEDULER_JOB --quiet\n",
"\n",
"# Delete Cloud Storage objects that were created.\n",
"! gsutil -m rm -r $PIPELINE_ROOT\n",
"! gsutil -m rm -r $TRAINING_ARTIFACTS_DIR"
]
"! gcloud storage rm --recursive $PIPELINE_ROOT\n", "! gcloud storage rm --recursive $TRAINING_ARTIFACTS_DIR" ]
}
],
"metadata": {
@@ -398,6 +398,7 @@
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
@@ -472,7 +473,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location $REGION $BUCKET_NAME"
]
},
{
@@ -492,7 +493,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -565,7 +566,7 @@
"outputs": [],
"source": [
"# Copy the sample data into your DATA_PATH\n",
"! gsutil cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $DATA_PATH"
"! gcloud storage cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $DATA_PATH"
]
},
{
@@ -579,11 +580,15 @@
"# Set hyperparameters.\n",
"BATCH_SIZE = 8 # @param {type:\"integer\"} Training and prediction batch size.\n",
"TRAINING_LOOPS = 5 # @param {type:\"integer\"} Number of training iterations.\n",
"STEPS_PER_LOOP = 2 # @param {type:\"integer\"} Number of driver steps per training iteration.\n",
"STEPS_PER_LOOP = (\n",
" 2 # @param {type:\"integer\"} Number of driver steps per training iteration.\n",
")\n",
"\n",
"# Set MovieLens simulation environment parameters.\n",
"RANK_K = 20 # @param {type:\"integer\"} Rank for matrix factorization in the MovieLens environment; also the observation dimension.\n",
"NUM_ACTIONS = 20 # @param {type:\"integer\"} Number of actions (movie items) to choose from.\n",
"NUM_ACTIONS = (\n",
" 20 # @param {type:\"integer\"} Number of actions (movie items) to choose from.\n",
")\n",
"PER_ARM = False # Use the non-per-arm version of the MovieLens environment.\n",
"\n",
"# Set agent parameters.\n",
@@ -621,7 +626,8 @@
"source": [
"# Define RL environment.\n",
"env = movielens_py_environment.MovieLensPyEnvironment(\n",
" DATA_PATH, RANK_K, BATCH_SIZE, num_movies=NUM_ACTIONS, csv_delimiter=\"\\t\")\n",
" DATA_PATH, RANK_K, BATCH_SIZE, num_movies=NUM_ACTIONS, csv_delimiter=\"\\t\"\n",
")\n",
"environment = tf_py_environment.TFPyEnvironment(env)\n",
"\n",
"# Define RL agent/algorithm.\n",
@@ -631,7 +637,8 @@
" tikhonov_weight=TIKHONOV_WEIGHT,\n",
" alpha=AGENT_ALPHA,\n",
" dtype=tf.float32,\n",
" accepts_per_arm_features=PER_ARM)\n",
" accepts_per_arm_features=PER_ARM,\n",
")\n",
"print(\"TimeStep Spec (for each batch):\\n\", agent.time_step_spec, \"\\n\")\n",
"print(\"Action Spec (for each batch):\\n\", agent.action_spec, \"\\n\")\n",
"print(\"Reward Spec (for each batch):\\n\", environment.reward_spec(), \"\\n\")\n",
@@ -639,7 +646,8 @@
"# Define RL metric.\n",
"optimal_reward_fn = functools.partial(\n",
" environment_utilities.compute_optimal_reward_with_movielens_environment,\n",
" environment=environment)\n",
" environment=environment,\n",
")\n",
"regret_metric = tf_bandit_metrics.RegretMetric(optimal_reward_fn)\n",
"metrics = [regret_metric]"
]
@@ -704,35 +712,38 @@
" if training_data_spec_transformation_fn is None:\n",
" data_spec = agent.policy.trajectory_spec\n",
" else:\n",
" data_spec = training_data_spec_transformation_fn(\n",
" agent.policy.trajectory_spec)\n",
" replay_buffer = trainer.get_replay_buffer(data_spec, environment.batch_size,\n",
" steps_per_loop)\n",
" data_spec = training_data_spec_transformation_fn(agent.policy.trajectory_spec)\n",
" replay_buffer = trainer.get_replay_buffer(\n",
" data_spec, environment.batch_size, steps_per_loop\n",
" )\n",
"\n",
" # `step_metric` records the number of individual rounds of bandit interaction;\n",
" # that is, (number of trajectories) * batch_size.\n",
" step_metric = tf_metrics.EnvironmentSteps()\n",
" metrics = [\n",
" tf_metrics.NumberOfEpisodes(),\n",
" tf_metrics.AverageEpisodeLengthMetric(batch_size=environment.batch_size)\n",
" tf_metrics.AverageEpisodeLengthMetric(batch_size=environment.batch_size),\n",
" ]\n",
" if additional_metrics:\n",
" metrics += additional_metrics\n",
"\n",
" if isinstance(environment.reward_spec(), dict):\n",
" metrics += [tf_metrics.AverageReturnMultiMetric(\n",
" reward_spec=environment.reward_spec(),\n",
" batch_size=environment.batch_size)]\n",
" else:\n",
" metrics += [\n",
" tf_metrics.AverageReturnMetric(batch_size=environment.batch_size)]\n",
" tf_metrics.AverageReturnMultiMetric(\n",
" reward_spec=environment.reward_spec(), batch_size=environment.batch_size\n",
" )\n",
" ]\n",
" else:\n",
" metrics += [tf_metrics.AverageReturnMetric(batch_size=environment.batch_size)]\n",
"\n",
" # Store intermediate metric results, indexed by metric names.\n",
" metric_results = defaultdict(list)\n",
"\n",
" if training_data_spec_transformation_fn is not None:\n",
" def add_batch_fn(data): return replay_buffer.add_batch(training_data_spec_transformation_fn(data)) \n",
" \n",
"\n",
" def add_batch_fn(data):\n",
" return replay_buffer.add_batch(training_data_spec_transformation_fn(data))\n",
"\n",
" else:\n",
" add_batch_fn = replay_buffer.add_batch\n",
"\n",
@@ -742,10 +753,12 @@
" env=environment,\n",
" policy=agent.collect_policy,\n",
" num_steps=steps_per_loop * environment.batch_size,\n",
" observers=observers)\n",
" observers=observers,\n",
" )\n",
"\n",
" training_loop = trainer.get_training_loop_fn(\n",
" driver, replay_buffer, agent, steps_per_loop)\n",
" driver, replay_buffer, agent, steps_per_loop\n",
" )\n",
" saver = policy_saver.PolicySaver(agent.policy)\n",
"\n",
" for _ in range(training_loops):\n",
@@ -783,7 +796,8 @@
" environment=environment,\n",
" training_loops=TRAINING_LOOPS,\n",
" steps_per_loop=STEPS_PER_LOOP,\n",
" additional_metrics=metrics)\n",
" additional_metrics=metrics,\n",
")\n",
"\n",
"tf.profiler.experimental.stop()"
]
@@ -1092,11 +1106,15 @@
},
"outputs": [],
"source": [
"RUN_HYPERPARAMETER_TUNING = True # Execute hyperparameter tuning instead of regular training.\n",
"RUN_HYPERPARAMETER_TUNING = (\n",
" True # Execute hyperparameter tuning instead of regular training.\n",
")\n",
"TRAIN_WITH_BEST_HYPERPARAMETERS = False # Do not train.\n",
"\n",
"HPTUNING_RESULT_DIR = \"hptuning/\" # @param {type: \"string\"} Directory to store the best hyperparameter(s) in `BUCKET_NAME` and locally (temporarily).\n",
"HPTUNING_RESULT_PATH = os.path.join(HPTUNING_RESULT_DIR, \"result.json\") # @param {type: \"string\"} Path to the file containing the best hyperparameter(s)."
"HPTUNING_RESULT_PATH = os.path.join(\n",
" HPTUNING_RESULT_DIR, \"result.json\"\n",
") # @param {type: \"string\"} Path to the file containing the best hyperparameter(s)."
]
},
{
@@ -1124,7 +1142,7 @@
" image_uri: str,\n",
" args: List[str],\n",
" location: str = \"us-central1\",\n",
" api_endpoint: str = \"us-central1-aiplatform.googleapis.com\"\n",
" api_endpoint: str = \"us-central1-aiplatform.googleapis.com\",\n",
") -> None:\n",
" \"\"\"Creates a hyperparameter tuning job using a custom container.\n",
"\n",
@@ -1197,8 +1215,8 @@
"\n",
" # Create job\n",
" response = client.create_hyperparameter_tuning_job(\n",
" parent=parent,\n",
" hyperparameter_tuning_job=hyperparameter_tuning_job)\n",
" parent=parent, hyperparameter_tuning_job=hyperparameter_tuning_job\n",
" )\n",
" job_id = response.name.split(\"/\")[-1]\n",
" print(\"Job ID:\", job_id)\n",
" print(\"Job config:\", response)\n",
@@ -1242,7 +1260,8 @@
" image_uri=f\"gcr.io/{PROJECT_ID}/{HPTUNING_TRAINING_CONTAINER}:latest\",\n",
" args=args,\n",
" location=REGION,\n",
" api_endpoint=f\"{REGION}-aiplatform.googleapis.com\")"
" api_endpoint=f\"{REGION}-aiplatform.googleapis.com\",\n",
")"
]
},
{
@@ -1292,7 +1311,8 @@
" name = client.hyperparameter_tuning_job_path(\n",
" project=project,\n",
" location=location,\n",
" hyperparameter_tuning_job=hyperparameter_tuning_job_id)\n",
" hyperparameter_tuning_job=hyperparameter_tuning_job_id,\n",
" )\n",
" response = client.get_hyperparameter_tuning_job(name=name)\n",
" return response"
]
@@ -1313,7 +1333,8 @@
" location=REGION,\n",
" api_endpoint=f\"{REGION}-aiplatform.googleapis.com\")\n",
" if response.state.name == 'JOB_STATE_SUCCEEDED':\n",
" print(\"Job succeeded.\\nJob Time:\", response.update_time - response.create_time)\n",
" print(\"Job succeeded.\n",
"Job Time:\", response.update_time - response.create_time)\n",
" trials = response.trials\n",
" print(\"Trials:\", trials)\n",
" break\n",
@@ -1348,8 +1369,8 @@
"if trials:\n",
" # Dict mapping from metric names to the best metric values seen so far\n",
" best_objective_values = dict.fromkeys(\n",
" [metric.metric_id for metric in trials[0].final_measurement.metrics],\n",
" -np.inf)\n",
" [metric.metric_id for metric in trials[0].final_measurement.metrics], -np.inf\n",
" )\n",
" # Dict mapping from metric names to a list of the best combination(s) of\n",
" # hyperparameter(s). Each combination is a dict mapping from hyperparameter\n",
" # names to their values.\n",
@@ -1358,12 +1379,13 @@
" # `final_measurement` and `parameters` are `RepeatedComposite` objects.\n",
" # Reference the structure above to extract the value of your interest.\n",
" for metric in trial.final_measurement.metrics:\n",
" params = {\n",
" param.parameter_id: param.value for param in trial.parameters}\n",
" params = {param.parameter_id: param.value for param in trial.parameters}\n",
" if metric.value > best_objective_values[metric.metric_id]:\n",
" best_params[metric.metric_id] = [params]\n",
" elif metric.value == best_objective_values[metric.metric_id]:\n",
" best_params[param.parameter_id].append(params) # Handle cases where multiple hyperparameter values lead to the same performance.\n",
" best_params[param.parameter_id].append(\n",
" params\n",
" ) # Handle cases where multiple hyperparameter values lead to the same performance.\n",
" print(\"Best hyperparameter value(s):\")\n",
" for metric, params in best_params.items():\n",
" print(f\"Metric={metric}: {sorted(params)}\")\n",
@@ -1443,7 +1465,9 @@
},
"outputs": [],
"source": [
"PREDICTION_CONTAINER = \"prediction-custom-container\" # @param {type:\"string\"} Name of the container image."
"PREDICTION_CONTAINER = (\n",
" \"prediction-custom-container\" # @param {type:\"string\"} Name of the container image.\n",
")"
]
},
{
@@ -1475,7 +1499,7 @@
" machineType: 'E2_HIGHCPU_8'\"\"\".format(\n",
" PROJECT_ID=PROJECT_ID,\n",
" PREDICTION_CONTAINER=PREDICTION_CONTAINER,\n",
" ARTIFACTS_DIR=ARTIFACTS_DIR\n",
" ARTIFACTS_DIR=ARTIFACTS_DIR,\n",
")\n",
"\n",
"with open(\"cloudbuild.yaml\", \"w\") as fp:\n",
@@ -1592,8 +1616,12 @@
},
"outputs": [],
"source": [
"RUN_HYPERPARAMETER_TUNING = False # Execute regular training instead of hyperparameter tuning.\n",
"TRAIN_WITH_BEST_HYPERPARAMETERS = True # @param {type:\"bool\"} Whether to use learned hyperparameters in training."
"RUN_HYPERPARAMETER_TUNING = (\n",
" False # Execute regular training instead of hyperparameter tuning.\n",
")\n",
"TRAIN_WITH_BEST_HYPERPARAMETERS = (\n",
" True # @param {type:\"bool\"} Whether to use learned hyperparameters in training.\n",
")"
]
},
{
@@ -1633,10 +1661,12 @@
"job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=\"train-movielens\",\n",
" container_uri=f\"gcr.io/{PROJECT_ID}/{HPTUNING_TRAINING_CONTAINER}:latest\",\n",
" command=[\"python3\", \"-m\", \"src.training.task\"] + args, # Pass in training arguments, including hyperparameters.\n",
" command=[\"python3\", \"-m\", \"src.training.task\"]\n",
" + args, # Pass in training arguments, including hyperparameters.\n",
" model_serving_container_image_uri=f\"gcr.io/{PROJECT_ID}/{PREDICTION_CONTAINER}:latest\",\n",
" model_serving_container_predict_route=\"/predict\",\n",
" model_serving_container_health_route=\"/health\")\n",
" model_serving_container_health_route=\"/health\",\n",
")\n",
"\n",
"print(\"Training Spec:\", job._managed_model)\n",
"\n",
@@ -1645,7 +1675,8 @@
" replica_count=1,\n",
" machine_type=\"n1-standard-4\",\n",
" accelerator_type=\"ACCELERATOR_TYPE_UNSPECIFIED\",\n",
" accelerator_count=0)"
" accelerator_count=0,\n",
")"
]
},
{
@@ -1784,7 +1815,7 @@
"! gcloud ai models delete $model.name --quiet\n",
"\n",
"# Delete Cloud Storage objects that were created\n",
"! gsutil -m rm -r $ARTIFACTS_DIR"
"! gcloud storage rm --recursive $ARTIFACTS_DIR"
]
}
],
@@ -324,7 +324,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -344,7 +344,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
}
],
@@ -328,7 +328,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -348,7 +348,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
}
],
@@ -341,7 +341,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -361,7 +361,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
}
],
@@ -2,9 +2,9 @@ dllogger@git+https://github.com/NVIDIA/dllogger@v1.0.0
# Fixing these libraries versions to avoid conflicting or broken packages.
immutabledict==4.2.1
protobuf==4.25.8
protobuf==5.29.6
opencv-python-headless==4.11.0.86
docutils==0.16
urllib3==2.5.0
urllib3==2.6.3
google-cloud-storage==3.0.0
retrying
@@ -31,8 +31,8 @@ packaging==25.0
pathspec==0.12.1
platformdirs==4.3.8
proto-plus==1.26.1
protobuf==5.29.4
pyasn1==0.6.1
protobuf==5.29.6
pyasn1==0.6.2
pyasn1_modules==0.4.2
pydantic==2.11.4
pydantic_core==2.33.2
@@ -45,5 +45,5 @@ six==1.17.0
sniffio==1.3.1
typing-inspection==0.4.0
typing_extensions==4.13.2
urllib3==2.5.0
urllib3==2.6.3
websockets==15.0.1
@@ -66,7 +66,7 @@ mkdir -p "$local_folder"
mkdir -p "$output_folder"
# Download the content from the GCS URI
gsutil -m cp -r "$gcs_dataset_path"/* "$local_folder/"
gcloud storage cp --recursive "$gcs_dataset_path"/* "$local_folder/"
# Process files in the local folder
for file in "$local_folder"/*; do
@@ -122,23 +122,23 @@ cp -r "$output_folder" "$images_folder"/images_2
pushd "$images_folder"/images_2
ls | xargs -P 8 -I {} mogrify -resize 50% {}
popd
gsutil -m cp -r "$images_folder"/images_2/* "$gcs_experiment_path"/data/images_2
gcloud storage cp --recursive "$images_folder"/images_2/* "$gcs_experiment_path"/data/images_2
cp -r "$output_folder" "$images_folder"/images_4
pushd "$images_folder"/images_4
ls | xargs -P 8 -I {} mogrify -resize 25% {}
popd
gsutil -m cp -r "$images_folder"/images_4/* "$gcs_experiment_path"/data/images_4
gcloud storage cp --recursive "$images_folder"/images_4/* "$gcs_experiment_path"/data/images_4
cp -r "$output_folder" "$images_folder"/images_8
pushd "$images_folder"/images_8
ls | xargs -P 8 -I {} mogrify -resize 12.5% {}
popd
gsutil -m cp "$images_folder"/images_8/* "$gcs_experiment_path"/data/images_8
gcloud storage cp "$images_folder"/images_8/* "$gcs_experiment_path"/data/images_8
# Copy images and sparse reconstruction files to gcs experiment folder.
gsutil -m cp "$images_folder"/images/* "$gcs_experiment_path"/data/images
gsutil -m cp -r "$local_folder"/sparse "$gcs_experiment_path"/data
gsutil -m cp "$local_folder"/database.db "$gcs_experiment_path"/data
gcloud storage cp "$images_folder"/images/* "$gcs_experiment_path"/data/images
gcloud storage cp --recursive "$local_folder"/sparse "$gcs_experiment_path"/data
gcloud storage cp "$local_folder"/database.db "$gcs_experiment_path"/data
echo "Processing complete."
@@ -99,14 +99,14 @@ create_dir_if_not_exists "$CHECKPOINTS_PATH"
touch "$local_experiment_path/$exp_folder_name/log_render.txt"
# Copy experiment from GCS bucket to local
gsutil -m cp -r "${args[-gcs_experiment_path]}/data" "$local_experiment_path/$exp_folder_name" || exit 1
gsutil -m cp -r "${args[-gcs_experiment_path]}/checkpoints/${training_job_name}/*" "$CHECKPOINTS_PATH" || exit 1
gcloud storage cp --recursive "${args[-gcs_experiment_path]}/data" "$local_experiment_path/$exp_folder_name" || exit 1
gcloud storage cp --recursive "${args[-gcs_experiment_path]}/checkpoints/${training_job_name}/*" "$CHECKPOINTS_PATH" || exit 1
# Check and copy keyframes file.
if [[ -n ${args[-gcs_keyframes_file]} ]]; then
keyframes_file_basename=$(basename "${args[-gcs_keyframes_file]}")
local_keyframes_file="$local_dataset_path/$keyframes_file_basename"
gsutil cp "${args[-gcs_keyframes_file]}" "$local_keyframes_file" || exit 1
gcloud storage cp "${args[-gcs_keyframes_file]}" "$local_keyframes_file" || exit 1
echo "Local keyframe file: $local_keyframes_file"
launch_rendering "$local_keyframes_file"
else
@@ -114,4 +114,4 @@ else
fi
# Copy rendered data back to GCS.
gsutil -m cp -r "$OUTPUT_RENDER_PATH" "${args[-gcs_experiment_path]}/render/${rendering_job_name}"
gcloud storage cp --recursive "$OUTPUT_RENDER_PATH" "${args[-gcs_experiment_path]}/render/${rendering_job_name}"
@@ -74,7 +74,7 @@ create_dir_if_not_exists "$local_experiment_path"
create_dir_if_not_exists "$local_experiment_path/$scene_folder_name"
# Copy experiment from GCS bucket to local.
gsutil -m cp -r "${gcs_experiment_path}/data" "$local_experiment_path/$scene_folder_name" || exit 1
gcloud storage cp --recursive "${gcs_experiment_path}/data" "$local_experiment_path/$scene_folder_name" || exit 1
echo "GCS Experiment: $gcs_experiment_path"
echo "Gin Config File: $gin_config_file"
@@ -89,6 +89,6 @@ accelerate launch train.py --gin_configs="$gin_config_file" \
--gin_bindings="Config.factor = ${factor}" \
--gin_bindings="Config.max_steps = ${max_training_steps}"
gsutil -m rm -r "${gcs_experiment_path}/checkpoints/${training_job_name}"
gsutil -m cp -r "$local_experiment_path/$scene_folder_name/config.gin" "${gcs_experiment_path}/${training_job_name}_config.gin"
gsutil -m cp -r "$local_experiment_path/$scene_folder_name/checkpoints/*/*" "${gcs_experiment_path}/checkpoints/${training_job_name}"
gcloud storage rm --recursive "${gcs_experiment_path}/checkpoints/${training_job_name}"
gcloud storage cp --recursive "$local_experiment_path/$scene_folder_name/config.gin" "${gcs_experiment_path}/${training_job_name}_config.gin"
gcloud storage cp --recursive "$local_experiment_path/$scene_folder_name/checkpoints/*/*" "${gcs_experiment_path}/checkpoints/${training_job_name}"
@@ -102,10 +102,10 @@ def download_gcs_uri_to_local(
if not os.path.exists(destination_dir):
os.mkdir(destination_dir)
subprocess.check_output([
"gsutil",
"-m",
"gcloud",
"storage",
"cp",
"-r",
"--recursive",
gcs_uri,
destination_dir,
])
+2
View File
@@ -27,6 +27,8 @@
/vertex_endpoints/nvidia-triton/nvidia-triton-custom-container-prediction.ipynb @RajeshThallam
/vertex_endpoints/optimized_tensorflow_runtime @vlasenkoalexey
/notebooks/community/alphagenome/cloudai_alphagenome_vai_quickstart.ipynb @dpanigra
/notebooks/community/alphagenome/cloudai_alphagenome_finetune.ipynb @dpanigra
/notebooks/community/weathernext/weathernext_2_early_access_program.ipynb @dpanigra
/notebooks/community/ml_ops/stage2/get_started_with_visionapi_and_automl.ipynb @mansari
/notebooks/community/neo4j/graph_paysim.ipynb @benofben @laeg
/notebooks/community/ml_ops/stage1/get_started_with_visionapi_and_vertex_datasets.ipynb @mansari
@@ -0,0 +1,552 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f705f4be70e9"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9a05a17e4e2f"
},
"source": [
"# Deploy ADK Agents to Vertex AI Agent Engine Using Inline Source\n",
"\n",
"<table><tbody><tr>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fagent_engine%2Fadk_inline_source_deployment.ipynb\">\n",
" <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/agent_engine/adk_inline_source_deployment.ipynb\">\n",
" <img alt=\"GitHub logo\" src=\"https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png\" width=\"32px\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</tr></tbody></table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e6e9014cd53a"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to deploy [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs/) agents to [Vertex AI Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview) using **inline source deployment**.\n",
"\n",
"I built this pattern while developing a multi-agent system with ADK. The approach is also featured in Google's [Agent Starter Pack community showcase](https://github.com/GoogleCloudPlatform/agent-starter-pack#community-showcase).\n",
"\n",
"Inline source deployment sends your agent's Python source code directly to Agent Engine, avoiding serialization issues that can occur with the default pickle-based deployment.\n",
"\n",
"### Why Inline Source Deployment?\n",
"\n",
"| Approach | How it works | Best for |\n",
"|----------|--------------|----------|\n",
"| Default (pickle) | Serializes agent object | Simple agents, quick prototypes |\n",
"| **Inline source** | Sends Python source directly | Complex agents, custom dependencies |\n",
"\n",
"Inline source deployment is useful when:\n",
"- Your agent uses custom classes that don't serialize well\n",
"- You need precise control over the deployment environment\n",
"- You want to avoid pickle compatibility issues across Python versions\n",
"\n",
"### Objectives\n",
"\n",
"- Create an ADK agent with tools\n",
"- Deploy to Agent Engine using inline source\n",
"- Test the deployed agent\n",
"- Clean up resources\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI Agent Engine\n",
"* Vertex AI (Gemini API)\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4d998a5140b2"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c832028166e3"
},
"source": [
"### Install required packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6b2e64d59bf4"
},
"outputs": [],
"source": [
"# Define requirements once for reuse\n",
"REQUIREMENTS = [\n",
" \"google-cloud-aiplatform[adk,agent_engines]>=1.88.0\",\n",
" \"google-adk>=1.18.0\",\n",
"]\n",
"\n",
"!pip install --upgrade --quiet {\" \".join(REQUIREMENTS)}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2f2e1483e7a8"
},
"source": [
"### Restart runtime\n",
"\n",
"After installing packages, restart the runtime to ensure the new packages are loaded."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d76628df8180"
},
"outputs": [],
"source": [
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "586675a64f57"
},
"source": [
"### Authenticate your notebook environment (Colab only)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2082bbcb771c"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "296a2345bc1a"
},
"source": [
"### Set Google Cloud project information"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "abafb08d10b1"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"STAGING_BUCKET = f\"gs://{PROJECT_ID}-agent-engine-staging\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c7863dcf6a38"
},
"source": [
"### Create staging bucket\n",
"\n",
"Agent Engine requires a staging bucket to store deployment artifacts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "17ece3f9fb6e"
},
"outputs": [],
"source": [
"!gsutil mb -l {LOCATION} -p {PROJECT_ID} {STAGING_BUCKET} 2>/dev/null || echo \"Bucket already exists\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1029f595561d"
},
"source": [
"### Initialize Vertex AI"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cac3c1b09203"
},
"outputs": [],
"source": [
"import vertexai\n",
"\n",
"vertexai.init(\n",
" project=PROJECT_ID,\n",
" location=LOCATION,\n",
" staging_bucket=STAGING_BUCKET,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "46fb887b5ea3"
},
"source": [
"## Create an ADK Agent\n",
"\n",
"First, let's create a simple ADK agent with a tool. This agent can answer questions and get the current time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9d0dd0808c69"
},
"outputs": [],
"source": [
"import datetime\n",
"from zoneinfo import ZoneInfo, ZoneInfoNotFoundError\n",
"\n",
"from google.adk.agents import LlmAgent\n",
"\n",
"\n",
"def get_current_time(timezone: str = \"UTC\") -> str:\n",
" \"\"\"Get the current time in the specified timezone.\n",
"\n",
" Args:\n",
" timezone: IANA timezone name (e.g., 'America/New_York', 'UTC')\n",
"\n",
" Returns:\n",
" Current time as a formatted string\n",
" \"\"\"\n",
" try:\n",
" tz = ZoneInfo(timezone)\n",
" now = datetime.datetime.now(tz)\n",
" return (\n",
" f\"The current time in {timezone} is {now.strftime('%Y-%m-%d %H:%M:%S %Z')}\"\n",
" )\n",
" except ZoneInfoNotFoundError:\n",
" return f\"Error: Timezone '{timezone}' not found. Please use a valid IANA timezone name (e.g., 'America/New_York', 'UTC').\"\n",
" except Exception as e:\n",
" return f\"An unexpected error occurred while getting time for timezone {timezone}: {e}\"\n",
"\n",
"\n",
"# Create the agent\n",
"agent = LlmAgent(\n",
" name=\"time_agent\",\n",
" model=\"gemini-1.5-flash-001\",\n",
" instruction=\"You are a helpful assistant. Use the get_current_time tool when asked about the time.\",\n",
" tools=[get_current_time],\n",
")\n",
"\n",
"print(f\"Created agent: {agent.name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ad0b9cc230ca"
},
"source": [
"### Test the agent locally\n",
"\n",
"Before deploying, let's test the agent locally to make sure it works."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1d3d71bde31e"
},
"outputs": [],
"source": [
"from google.adk import Runner\n",
"from google.adk.sessions import InMemorySessionService\n",
"from google.genai import types\n",
"\n",
"# Create a local runner\n",
"runner = Runner(\n",
" agent=agent,\n",
" app_name=\"test_app\",\n",
" session_service=InMemorySessionService(),\n",
")\n",
"\n",
"# Create a session and test\n",
"session = runner.session_service.create_session(\n",
" app_name=\"test_app\",\n",
" user_id=\"test_user\",\n",
")\n",
"\n",
"# Send a test message\n",
"response = runner.run(\n",
" user_id=\"test_user\",\n",
" session_id=session.id,\n",
" new_message=types.Content(\n",
" role=\"user\",\n",
" parts=[types.Part(text=\"What time is it in New York?\")],\n",
" ),\n",
")\n",
"\n",
"# Print the response\n",
"for event in response:\n",
" if hasattr(event, \"content\") and event.content:\n",
" for part in event.content.parts:\n",
" if hasattr(part, \"text\") and part.text:\n",
" print(part.text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "06bdc3c10ea6"
},
"source": [
"## Deploy Using Inline Source\n",
"\n",
"Now let's deploy the agent to Agent Engine using inline source deployment.\n",
"\n",
"The key difference from default deployment is using `AgentEngineApp.from_agent()` which packages the source code directly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d641a5e29291"
},
"outputs": [],
"source": [
"from vertexai import agent_engines\n",
"from vertexai.agent_engines import AgentEngineApp\n",
"\n",
"# Create the app from the agent (inline source)\n",
"app = AgentEngineApp.from_agent(\n",
" agent=agent,\n",
" enable_tracing=True,\n",
")\n",
"\n",
"print(\"App created successfully\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cfd8fddbf181"
},
"source": [
"### Deploy to Agent Engine"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ebd6642375a8"
},
"outputs": [],
"source": [
"# Deploy the app\n",
"# This may take a few minutes\n",
"remote_app = agent_engines.create(\n",
" app,\n",
" requirements=REQUIREMENTS,\n",
" display_name=\"time-agent-inline\",\n",
")\n",
"\n",
"print(f\"Deployed agent: {remote_app.resource_name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "43b739d2f980"
},
"source": [
"## Test the Deployed Agent\n",
"\n",
"Now let's test the deployed agent by sending it a query."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8ec0598a1138"
},
"outputs": [],
"source": [
"# Create a session with the remote agent\n",
"remote_session = remote_app.create_session(user_id=\"test_user\")\n",
"\n",
"print(f\"Created session: {remote_session['id']}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f85603f0e4fd"
},
"outputs": [],
"source": [
"# Query the agent\n",
"response = remote_app.stream_query(\n",
" user_id=\"test_user\",\n",
" session_id=remote_session[\"id\"],\n",
" message=\"What time is it in Tokyo?\",\n",
")\n",
"\n",
"# Print the response\n",
"for chunk in response:\n",
" if hasattr(chunk, \"content\") and chunk.content:\n",
" for part in chunk.content.parts:\n",
" if hasattr(part, \"text\") and part.text:\n",
" print(part.text, end=\"\")\n",
"\n",
"print() # Newline at end"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e70d97acf3a3"
},
"source": [
"## Cleaning up\n",
"\n",
"Delete the deployed agent to avoid ongoing charges."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6d2d34a6ba7c"
},
"outputs": [],
"source": [
"# Delete the deployed agent\n",
"delete_agent = False # @param {type:\"boolean\"}\n",
"\n",
"if delete_agent:\n",
" remote_app.delete()\n",
" print(\"Agent deleted successfully\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "35870ecf063a"
},
"outputs": [],
"source": [
"# Delete the staging bucket\n",
"delete_bucket = False # @param {type:\"boolean\"}\n",
"\n",
"if delete_bucket:\n",
" !gsutil rm -r {STAGING_BUCKET}\n",
" print(\"Bucket deleted successfully\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "11f2cc08759a"
},
"source": [
"## Summary\n",
"\n",
"In this notebook, you learned how to:\n",
"\n",
"1. Create an ADK agent with tools\n",
"2. Test the agent locally\n",
"3. Deploy to Agent Engine using inline source with `AgentEngineApp.from_agent()`\n",
"4. Query the deployed agent\n",
"\n",
"### Next steps\n",
"\n",
"- Learn more about [ADK](https://google.github.io/adk-docs/)\n",
"- Explore [Agent Engine documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview)\n",
"- Try adding more complex tools and multi-agent orchestration"
]
}
],
"metadata": {
"colab": {
"name": "adk_inline_source_deployment.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
+6 -2
View File
@@ -1,7 +1,8 @@
![AlphaGenome header image](https://raw.githubusercontent.com/google-deepmind/alphagenome/refs/heads/main/docs/source/_static/header.png)
# AlphaGenome
[**Overview**](#overview) | [**Use Cases**](#use-cases) | [**Documentation**](#documentation) | [**Pricing**](#pricing) | [**Quick start**](#quick-start)
[**Overview**](#overview) | [**Use Cases**](#use-cases) | [**Documentation**](#documentation) | [**Pricing**](#pricing) | [**Quick start inference**](#quick-start-inference) |
[**Quick start finetune**](#quick-start-finetune)
## Overview
**Disclaimer:** *Experimental*.
@@ -89,5 +90,8 @@ To utilize these models via this service:
* **Pricing information** will be shared directly with users upon approval
and placement on the allowlist.
## Quick start
## Quick start inference
The quickest way to get started with the AlphaGenome in Google Cloud Platform is to run [our example notebook](cloudai_alphagenome_vai_quickstart.ipynb) in [Google Colab](https://colab.research.google.com/).
## Quick start finetune
The quickest way to get started with the AlphaGenome fineutning in Google Cloud Platform is to run [our finetuning notebook](cloudai_alphagenome_finetune.ipynb) in [Google Cloud Platform Enterprise Colab](https://docs.cloud.google.com/colab/docs/introduction).
File diff suppressed because it is too large Load Diff
@@ -37,7 +37,7 @@
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2alphagenome%2Fcloudai_alphagenome_vai_quickstart.ipynb\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Falphagenome%2Fcloudai_alphagenome_vai_quickstart.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
@@ -48,7 +48,7 @@
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/alphagenome/cloudai_alphagenome_vai_quickstart.ipynb\">\n",
" <img width=\"32px\"src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" <img width=\"32px\"src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
@@ -91,7 +91,7 @@
"id": "FUmKwF5ZMMS-"
},
"source": [
"## Authenticate your notebook environment (Colab only)"
"## Authenticate (option 1) your notebook environment (Colab only)"
]
},
{
@@ -129,14 +129,40 @@
},
"outputs": [],
"source": [
"gcp_project = \"\" # @param {type:\"string\"}\n",
"vertex_ai_url = \"\" # @param {type:\"string\"}\n",
"service_account_key = \"\" # @param {type:\"string\"}\n",
"# Keep Service account and token empty when using the key\n",
"service_account = \"\" # @param {type:\"string\"}\n",
"token = \"\" # @param {type:\"string\"}\n",
"code_whl = \"alphagenome-0.4.2.2-py3-none-any.whl\"\n",
"code_whl = \"alphagenome-0.4.2.6-py3-none-any.whl\"\n",
"file_path = f\"gs://alphagenome-whl/{code_whl}\"\n",
"\n",
"service_account = service_account or None\n",
"token = token or None\n"
"token = token or None"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1902445c3133"
},
"source": [
"## Authenticate (option 2) using your service account key"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1ed156e7c578"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"!gcloud auth activate-service-account --key-file={service_account_key}\n",
"os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = service_account_key"
]
},
{
@@ -170,7 +196,7 @@
" print(f\"Service Account is set to '{service_account}'. Proceeding with the copy...\")\n",
" !gcloud config set auth/impersonate_service_account $service_account\n",
"print(f\"Downlaoding the wheel file: {file_path}\")\n",
"!gcloud storage cp {file_path} .\n",
"!gcloud storage cp {file_path} . --billing-project={gcp_project}\n",
"\n",
"# to unset impersonate_service_account\n",
"# !gcloud config unset auth/impersonate_service_account"
@@ -273,7 +299,6 @@
"print(\"Creating HttpDnaClient...\")\n",
"dna_model = dna_client_http.create_http_client(\n",
" vertex_ai_url=vertex_ai_url,\n",
" model_version=\"FOLD_0\",\n",
" service_account=service_account,\n",
" token=token,\n",
")\n",
@@ -4433,17 +4458,7 @@
"outputs": [],
"source": [
"def extract_k562(adata):\n",
" if \"ontologyTerm\" in adata.var.columns and not adata.var[\"ontologyTerm\"].empty:\n",
" mask = adata.var[\"ontologyTerm\"].apply(\n",
" lambda x: isinstance(x, dict)\n",
" and x.get(\"ontologyType\") == \"ONTOLOGY_TYPE_EFO\"\n",
" and x.get(\"id\") == \"2067\"\n",
" )\n",
" values = adata.X[:, mask]\n",
" else:\n",
" raise ValueError(\n",
" \"Expected 'ontologyTerm' column with dictionary values not found in adata.var\"\n",
" )\n",
" values = adata.X[:, adata.var[\"ontology_curie\"] == \"EFO:0002067\"]\n",
" assert values.size == 1\n",
" return values.flatten()[0]\n",
"\n",
@@ -4622,9 +4637,7 @@
}
],
"source": [
"interval = Interval(\n",
" chromosome=\"chr19\", start=2**20, end=2**20 + 2**20, strand=\".\"\n",
")\n",
"interval = Interval(chromosome=\"chr19\", start=2**20, end=2**20 + 2**20, strand=\".\")\n",
"\n",
"scores = dna_model.score_interval(\n",
" interval,\n",
@@ -42,7 +42,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/bigquery_ml/Anomaly_detection_in_Cloud_Audit_logs_with_BQML.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -52,7 +52,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/bigquery_ml/bq_ml_with_vision_translation_nlp.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -40,7 +40,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery_ml/bqml-online-prediction.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -359,7 +359,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
"! gcloud storage buckets create --location={REGION} --project={PROJECT_ID} {BUCKET_URI}"
]
},
{
@@ -1098,7 +1098,7 @@
" ! bq rm -r -f $PROJECT_ID:$BQ_DATASET_NAME\n",
"# delete the Cloud Storage bucket\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil -m rm -r $BUCKET_URI"
" ! gcloud storage rm --recursive $BUCKET_URI"
]
}
],
@@ -38,7 +38,7 @@
" </td>\n",
" <td>\n",
" <a href=\"github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/cohere/cohere_embedding_with_matching_engine.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -449,7 +449,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -470,7 +470,7 @@
"outputs": [],
"source": [
"# this will not return anything if the bucket is empty\n",
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -588,7 +588,7 @@
"# NOTE: Everything in this GCS DIR will be DELETED before uploading the data.\n",
"# A CommandException is expected if no data is present\n",
"\n",
"! gsutil rm -rf {BUCKET_NAME}/*"
"! gcloud storage rm --recursive --continue-on-error {BUCKET_NAME}/*"
]
},
{
@@ -599,7 +599,7 @@
},
"outputs": [],
"source": [
"! gsutil cp cohere_embeddings.json {BUCKET_NAME}/cohere_embeddings.json"
"! gcloud storage cp cohere_embeddings.json {BUCKET_NAME}/cohere_embeddings.json"
]
},
{
@@ -610,7 +610,7 @@
},
"outputs": [],
"source": [
"! gsutil ls {BUCKET_NAME}"
"! gcloud storage ls {BUCKET_NAME}"
]
},
{
@@ -41,7 +41,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/vertex_ai_experiments_classification.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -537,7 +537,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI"
]
},
{
@@ -557,7 +557,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
"! gcloud storage ls --all-versions --long $BUCKET_URI"
]
},
{
@@ -627,9 +627,9 @@
},
"outputs": [],
"source": [
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
"! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectCreator\n",
"\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
"! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectViewer"
]
},
{
@@ -1736,7 +1736,7 @@
"delete_bucket = False\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}"
]
}
],
@@ -38,7 +38,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/explainable_ai/SDK_Custom_Container_XAI.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -405,7 +405,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI"
]
},
{
@@ -425,7 +425,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
"! gcloud storage ls --all-versions --long $BUCKET_URI"
]
},
{
@@ -698,7 +698,7 @@
},
"outputs": [],
"source": [
"!gsutil cp app/model.joblib {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/"
"!gcloud storage cp app/model.joblib {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/"
]
},
{
@@ -1516,7 +1516,7 @@
"\n",
"delete_bucket = False\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}"
]
}
],
@@ -637,20 +637,20 @@
},
"outputs": [],
"source": [
"# Upload CSV data to Cloud Storage by passing gsutil commands to system\n",
"# Upload CSV data to Cloud Storage by passing gcloud storage commands to system\n",
"gcs_url <- paste0(\"gs://\", BUCKET_NAME, \"/\")\n",
"\n",
"command <- paste(\"gsutil mb\", gcs_url)\n",
"command <- paste(\"gcloud storage buckets create\", gcs_url)\n",
"\n",
"system(command)\n",
"\n",
"gcs_data_dir <- paste0(\"gs://\", BUCKET_NAME, \"/data\")\n",
"\n",
"command <- paste(\"gsutil cp data/*_data.csv\", gcs_data_dir)\n",
"command <- paste(\"gcloud storage cp data/*_data.csv\", gcs_data_dir)\n",
"\n",
"system(command)\n",
"\n",
"command <- paste(\"gsutil ls -l\", gcs_data_dir)\n",
"command <- paste(\"gcloud storage ls --long\", gcs_data_dir)\n",
"\n",
"system(command, intern = TRUE)"
]
@@ -679,4 +679,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}
@@ -38,7 +38,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/feature_store/gapic-feature-store.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,7 +34,7 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_vertex_feature_store_serving.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -471,8 +471,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
@@ -491,8 +490,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
@@ -511,8 +509,7 @@
},
"outputs": [],
"source": [
"! gsutil uniformbucketlevelaccess set on {BUCKET_URI}"
]
"! gcloud storage buckets update --uniform-bucket-level-access {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
@@ -1910,9 +1907,7 @@
},
"outputs": [],
"source": [
"! gsutil -m rm -r $BUCKET_URI\n",
"! gsutil rb $BUCKET_URI"
]
"! gcloud storage rm --recursive $BUCKET_URI\n", "! gcloud storage buckets delete $BUCKET_URI" ]
}
],
"metadata": {
@@ -38,7 +38,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/mobile_gaming/mobile_gaming_feature_store.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -37,7 +37,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/sdk-feature-store.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,7 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -441,7 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -478,9 +478,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -886,11 +884,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1329,7 +1327,7 @@
},
"outputs": [],
"source": [
"test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n2\n",
"if len(str(test_items[0]).split(\",\")) == 3:\n",
" _, test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
" _, test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
@@ -1363,8 +1361,8 @@
"file_1 = test_item_1.split(\"/\")[-1]\n",
"file_2 = test_item_2.split(\"/\")[-1]\n",
"\n",
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
"! gcloud storage cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gcloud storage cp $test_item_2 $BUCKET_NAME/$file_2\n",
"\n",
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
@@ -1408,7 +1406,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
"! gcloud storage cat $gcs_input_uri"
]
},
{
@@ -1692,8 +1690,8 @@
"outputs": [],
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" \"\"\"Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n",
" latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
@@ -1711,9 +1709,9 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n",
" break\n",
" time.sleep(60)"
]
@@ -1808,7 +1806,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_export_edge.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -416,7 +416,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -436,7 +436,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -474,7 +474,6 @@
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
@@ -784,11 +783,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1215,7 +1214,6 @@
" }\n",
" response = clients[\"model\"].export_model(name=name, output_config=output_config)\n",
" print(\"Long running operation:\", response.operation.name)\n",
" result = response.result(timeout=1800)\n",
" metadata = response.operation.metadata\n",
" artifact_uri = str(metadata.value).split(\"\\\\\")[-1][4:-1]\n",
" print(\"Artifact Uri\", artifact_uri)\n",
@@ -1244,9 +1242,9 @@
},
"outputs": [],
"source": [
"! gsutil ls $model_package\n",
"! gcloud storage ls $model_package\n",
"# Download the model artifacts\n",
"! gsutil cp -r $model_package tflite\n",
"! gcloud storage cp --recursive $model_package tflite\n",
"\n",
"tflite_path = \"tflite/model.tflite\""
]
@@ -1305,7 +1303,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n1\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"test_item = test_items[0].split(\",\")[0]\n",
"\n",
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
@@ -1449,7 +1447,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -721,12 +721,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1342,8 +1340,7 @@
},
"outputs": [],
"source": [
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
"if len(str(test_item[0]).split(\",\")) == 3:\n",
"test_item = !gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(str(test_item[0]).split(\",\")) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label = str(test_item[0]).split(\",\")\n",
@@ -1568,8 +1565,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_online_proxy.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -721,12 +721,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1342,8 +1340,7 @@
},
"outputs": [],
"source": [
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
"if len(str(test_item[0]).split(\",\")) == 3:\n",
"test_item = !gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(str(test_item[0]).split(\",\")) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label = str(test_item[0]).split(\",\")\n",
@@ -1747,8 +1744,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,7 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -441,7 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -478,9 +478,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -887,11 +885,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1333,7 +1331,7 @@
},
"outputs": [],
"source": [
"test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n2\n",
"cols_1 = str(test_items[0]).split(\",\")\n",
"cols_2 = str(test_items[1]).split(\",\")\n",
"if len(cols_1) == 11:\n",
@@ -1373,8 +1371,8 @@
"file_1 = test_item_1.split(\"/\")[-1]\n",
"file_2 = test_item_2.split(\"/\")[-1]\n",
"\n",
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
"! gcloud storage cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gcloud storage cp $test_item_2 $BUCKET_NAME/$file_2\n",
"\n",
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
@@ -1418,7 +1416,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
"! gcloud storage cat $gcs_input_uri"
]
},
{
@@ -1704,8 +1702,8 @@
"outputs": [],
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" \"\"\"Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n",
" latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
@@ -1723,9 +1721,9 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n",
" break\n",
" time.sleep(60)"
]
@@ -1820,7 +1818,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_export_edge.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -416,7 +416,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location $REGION $BUCKET_NAME"
]
},
{
@@ -436,7 +436,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -473,9 +473,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format\n"
]
},
{
@@ -785,11 +783,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1218,7 +1216,6 @@
" }\n",
" response = clients[\"model\"].export_model(name=name, output_config=output_config)\n",
" print(\"Long running operation:\", response.operation.name)\n",
" result = response.result(timeout=1800)\n",
" metadata = response.operation.metadata\n",
" artifact_uri = str(metadata.value).split(\"\\\\\")[-1][4:-1]\n",
" print(\"Artifact Uri\", artifact_uri)\n",
@@ -1247,9 +1244,9 @@
},
"outputs": [],
"source": [
"! gsutil ls $model_package\n",
"! gcloud storage ls $model_package\n",
"# Download the model artifacts\n",
"! gsutil cp -r $model_package tflite\n",
"! gcloud storage cp --recursive $model_package tflite\n",
"\n",
"tflite_path = \"tflite/model.tflite\""
]
@@ -1308,7 +1305,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n1\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"test_item = test_items[0].split(\",\")[0]\n",
"\n",
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
@@ -1452,7 +1449,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -396,9 +396,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -722,11 +720,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1345,7 +1343,7 @@
},
"outputs": [],
"source": [
"test_items = !gsutil cat $IMPORT_FILE | head -n1\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n1\n",
"cols = str(test_items[0]).split(\",\")\n",
"if len(cols) == 11:\n",
" test_item = str(cols[1])\n",
@@ -1574,7 +1572,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_segmentation_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -891,12 +889,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1325,8 +1321,7 @@
"source": [
"import json\n",
"\n",
"test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
"test_data_1 = test_items[0].replace(\"'\", '\"')\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n2\n", "test_data_1 = test_items[0].replace(\"'\", '\"')\n",
"test_data_1 = json.loads(test_data_1)\n",
"test_data_2 = test_items[0].replace(\"'\", '\"')\n",
"test_data_2 = json.loads(test_data_2)\n",
@@ -1367,9 +1362,8 @@
"file_1 = test_item_1.split(\"/\")[-1]\n",
"file_2 = test_item_2.split(\"/\")[-1]\n",
"\n",
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
"\n",
"! gcloud storage cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gcloud storage cp $test_item_2 $BUCKET_NAME/$file_2\n", "\n",
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
]
@@ -1412,8 +1406,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1681,8 +1674,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1699,10 +1691,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1796,8 +1786,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_segmentation_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -726,12 +726,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1338,8 +1336,7 @@
"source": [
"import json\n",
"\n",
"test_items = !gsutil cat $IMPORT_FILE | head -n1\n",
"test_data = test_items[0].replace(\"'\", '\"')\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n1\n", "test_data = test_items[0].replace(\"'\", '\"')\n",
"test_data = json.loads(test_data)\n",
"try:\n",
" test_item = test_data[\"image_gcs_uri\"]\n",
@@ -1555,8 +1552,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_binary_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -809,14 +807,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1377,8 +1372,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1646,8 +1640,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1664,10 +1657,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.csv\n",
"\n",
" ! gsutil cat $folder/prediction*.csv\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.csv\n", "\n",
" ! gcloud storage cat $folder/prediction*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1761,8 +1752,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_binary_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -735,14 +735,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1668,8 +1665,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -810,14 +808,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1363,8 +1358,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1632,8 +1626,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1650,10 +1643,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.csv\n",
"\n",
" ! gsutil cat $folder/prediction*.csv\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.csv\n", "\n",
" ! gcloud storage cat $folder/prediction*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1747,8 +1738,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_batch_explain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -810,14 +808,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1363,8 +1358,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1631,8 +1625,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1649,10 +1642,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/explanation*.csv\n",
"\n",
" ! gsutil cat $folder/explanation*.csv\n",
" break\n",
" ! gcloud storage ls $folder/explanation*.csv\n", "\n",
" ! gcloud storage cat $folder/explanation*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1746,8 +1737,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_export_cloud.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -416,8 +416,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -436,8 +435,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -749,14 +747,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1283,15 +1278,12 @@
"source": [
"print(\"Model Package:\", model_package)\n",
"print(\"Contents:\")\n",
"! gsutil ls $model_package\n",
"\n",
"! gcloud storage ls $model_package\n", "\n",
"print(\"\\nTF Saved Model\")\n",
"path = model_package + \"/predict\"\n",
"files = ! gsutil ls $path\n",
"saved_dir = files[1]\n",
"files = ! gcloud storage ls $path\n", "saved_dir = files[1]\n",
"print(saved_dir)\n",
"! gsutil ls $saved_dir"
]
"! gcloud storage ls $saved_dir" ]
},
{
"cell_type": "markdown",
@@ -1312,8 +1304,7 @@
},
"outputs": [],
"source": [
"! gsutil cp -r $model_package ."
]
"! gcloud storage cp --recursive $model_package ." ]
},
{
"cell_type": "markdown",
@@ -1599,8 +1590,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -735,14 +735,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1645,8 +1642,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_online_explain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -397,7 +397,6 @@
"\n",
"import google.cloud.aiplatform_v1beta1 as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
@@ -735,13 +734,13 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"! gcloud storage cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
@@ -1820,7 +1819,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_forecasting_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -779,14 +777,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = \"deaths\" # @param {type:\"string\"}\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = \"deaths\" # @param {type:\"string\"}\n",
"time_column = \"date\" # @param {type:\"string\"}\n",
"time_series_identifier_column = \"county\" # @param {type:\"string\"}\n",
"print(\"Label Column Name\", label_column)\n",
@@ -1358,8 +1353,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1627,8 +1621,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1645,10 +1638,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.csv\n",
"\n",
" ! gsutil cat $folder/prediction*.csv\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.csv\n", "\n",
" ! gcloud storage cat $folder/prediction*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1742,8 +1733,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_regression_online_bq.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -397,7 +397,6 @@
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
@@ -1614,7 +1613,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -881,12 +879,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1314,8 +1310,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"if len(test_items[0]) == 3:\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "if len(test_items[0]) == 3:\n",
" _, test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
" _, test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
"else:\n",
@@ -1371,8 +1366,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1644,8 +1638,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1662,10 +1655,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1759,8 +1750,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -716,12 +716,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1327,7 +1325,7 @@
},
"outputs": [],
"source": [
"test_item = ! gsutil cat $IMPORT_FILE | head -n1\n",
"test_item = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"if len(test_item[0]) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
@@ -1533,8 +1531,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_entity_extraction_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,7 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location $REGION $BUCKET_NAME"
]
},
{
@@ -441,7 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -478,9 +478,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -888,11 +886,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1377,7 +1375,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
"! gcloud storage cat $gcs_input_uri"
]
},
{
@@ -1652,8 +1650,8 @@
"outputs": [],
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" \"\"\"Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n",
" latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
@@ -1671,9 +1669,9 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n",
" break\n",
" time.sleep(60)"
]
@@ -1768,7 +1766,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_entity_extraction_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -723,12 +723,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1544,8 +1542,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_multi-label_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -881,12 +879,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1314,8 +1310,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"cols_1 = str(test_items[0]).split(\",\")\n",
"cols_2 = str(test_items[1]).split(\",\")\n",
"test_item_1 = cols_1[0]\n",
@@ -1372,8 +1367,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1645,8 +1639,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1663,10 +1656,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1760,8 +1751,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_multi-label_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -716,12 +716,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1327,8 +1325,7 @@
},
"outputs": [],
"source": [
"test_item = ! gsutil cat $IMPORT_FILE | head -n1\n",
"\n",
"test_item = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "\n",
"cols = str(test_item[0]).split(\",\")\n",
"test_item = cols[0]\n",
"test_label = cols[1:]\n",
@@ -1533,8 +1530,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_sentiment_analysis_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -883,12 +881,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1316,8 +1312,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"if len(test_items[0]) == 4:\n",
" _, test_item_1, test_label_1, _ = str(test_items[0]).split(\",\")\n",
" _, test_item_2, test_label_2, _ = str(test_items[1]).split(\",\")\n",
@@ -1375,8 +1370,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1643,8 +1637,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1661,10 +1654,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1758,8 +1749,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_sentiment_analysis_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -718,12 +718,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1329,8 +1327,7 @@
},
"outputs": [],
"source": [
"test_item = ! gsutil cat $IMPORT_FILE | head -n1\n",
"if len(test_item[0]) == 3:\n",
"test_item = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(test_item[0]) == 3:\n",
" _, test_item, test_label, max = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label, max = str(test_item[0]).split(\",\")\n",
@@ -1534,8 +1531,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_action_recognition_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -420,8 +420,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -440,8 +439,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -888,12 +886,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1310,8 +1306,7 @@
"import json\n",
"\n",
"import_file = IMPORT_FILES[0]\n",
"test_items = ! gsutil cat $import_file | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $import_file | head -n2\n", "\n",
"cols = str(test_items[0]).split(',')\n",
"test_item_1 = str(cols[0])\n",
"test_label_1 = str(cols[-1])\n",
@@ -1360,8 +1355,7 @@
" f.write(json.dumps(data) + '\\n')\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1637,8 +1631,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split('/')[-2]\n",
" if subfolder.startswith('prediction-'):\n",
@@ -1655,10 +1648,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1752,8 +1743,7 @@
" print(e)\n",
"\n",
"if delete_bucket and 'BUCKET_NAME' in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -420,8 +420,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -440,8 +439,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -885,12 +883,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1301,8 +1297,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"if len(test_items[0]) == 5:\n",
" _, test_item_1, test_label_1, _, _ = str(test_items[0]).split(',')\n",
" _, test_item_2, test_label_2, _, _ = str(test_items[1]).split(',')\n",
@@ -1351,8 +1346,7 @@
" f.write(json.dumps(data) + '\\n')\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1630,8 +1624,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split('/')[-2]\n",
" if subfolder.startswith('prediction-'):\n",
@@ -1648,10 +1641,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
"! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1745,8 +1736,7 @@
" print(e)\n",
"\n",
"if delete_bucket and 'BUCKET_NAME' in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_object_tracking_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -420,8 +420,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -440,8 +439,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -890,12 +888,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1309,8 +1305,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"cols_1 = test_items[0].split(',')\n",
"cols_2 = test_items[1].split(',')\n",
"if len(cols_1) > 12:\n",
@@ -1365,8 +1360,7 @@
" f.write(json.dumps(data) + '\\n')\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1638,8 +1632,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split('/')[-2]\n",
" if subfolder.startswith('prediction-'):\n",
@@ -1656,10 +1649,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1753,8 +1744,7 @@
" print(e)\n",
"\n",
"if delete_bucket and 'BUCKET_NAME' in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_batch_explain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_ab_testing.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_container.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -40,7 +40,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain_example_based_api.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_exported_ds.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_pipeline.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_raw.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_tfserving.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_super_resolution_online_post.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_classification_online_exported_ds.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_batch_explain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_container.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_explain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_pipeline.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_tfserving.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_batch.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_container.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_pipeline.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_tfserving.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_classification_online_exported_ds.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -426,8 +426,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -446,8 +445,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -483,9 +481,6 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
{
@@ -991,12 +986,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1153,8 +1146,7 @@
"source": [
"jsonl_index = result.exported_files[0]\n",
"\n",
"! gsutil cat $jsonl_index | head"
]
"! gcloud storage cat $jsonl_index | head" ]
},
{
"cell_type": "markdown",
@@ -1286,8 +1278,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_image_classification.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -419,8 +419,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -439,8 +438,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1119,8 +1117,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -1617,8 +1614,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -2062,8 +2058,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_tabular_regression.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -419,8 +419,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -439,8 +438,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1116,8 +1114,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -1626,8 +1623,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -2085,8 +2081,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_text_binary_classification.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -419,8 +419,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -439,8 +438,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1116,8 +1114,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -1618,8 +1615,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -2060,8 +2056,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -39,7 +39,7 @@
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_local_image_classification_online.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -423,8 +423,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -443,8 +442,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1718,8 +1716,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {

Some files were not shown because too many files have changed in this diff Show More