xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

use dataset instead of datasets (#892)

Browse Source 
less chance for an error and confusion in name clashing with the `datasets` pypi package also used in the notebook.

Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Andrew Maguire
2022-08-31 07:44:46 -07:00
committed by GitHub
co-authored by Andrew Ferlitsch
parent 96be449c69
commit c030d7ef74
1 changed files with 13 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
community-content/pytorch_text_classification_using_vertex_sdk_and_gcloud/pytorch-text-classification-vertex-ai-train-tune-deploy.ipynb
+13 -13
View File
@@ -658,8 +658,8 @@
},
"outputs": [],
"source": [
"datasets = load_dataset(\"imdb\")\n",
"datasets"
"dataset = load_dataset(\"imdb\")\n",
"dataset"
]
},
{
@@ -668,7 +668,7 @@
"id": "RzfPtOMoIrIu"
},
"source": [
"The `datasets` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set."
"The `dataset` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set."
]
},
{
@@ -681,12 +681,12 @@
"source": [
"print(\n",
" \"Total # of rows in training dataset {} and size {:5.2f} MB\".format(\n",
" datasets[\"train\"].shape[0], datasets[\"train\"].size_in_bytes / (1024 * 1024)\n",
" dataset[\"train\"].shape[0], dataset[\"train\"].size_in_bytes / (1024 * 1024)\n",
" )\n",
")\n",
"print(\n",
" \"Total # of rows in test dataset {} and size {:5.2f} MB\".format(\n",
" datasets[\"test\"].shape[0], datasets[\"test\"].size_in_bytes / (1024 * 1024)\n",
" dataset[\"test\"].shape[0], dataset[\"test\"].size_in_bytes / (1024 * 1024)\n",
" )\n",
")"
]
@@ -708,7 +708,7 @@
},
"outputs": [],
"source": [
"datasets[\"train\"][0]"
"dataset[\"train\"][0]"
]
},
{
@@ -728,7 +728,7 @@
},
"outputs": [],
"source": [
"label_list = datasets[\"train\"].unique(\"label\")\n",
"label_list = dataset[\"train\"].unique(\"label\")\n",
"label_list"
]
},
@@ -779,7 +779,7 @@
},
"outputs": [],
"source": [
"show_random_elements(datasets[\"train\"])"
"show_random_elements(dataset[\"train\"])"
]
},
{
@@ -883,7 +883,7 @@
},
"outputs": [],
"source": [
"example = datasets[\"train\"][4]\n",
"example = dataset[\"train\"][4]\n",
"print(example)"
]
},
@@ -920,7 +920,7 @@
"source": [
"# Dataset loading repeated here to make this cell idempotent\n",
"# Since we are over-writing datasets variable\n",
"datasets = load_dataset(\"imdb\")\n",
"dataset = load_dataset(\"imdb\")\n",
"\n",
"# Mapping labels to ids\n",
"# NOTE: We can extract this automatically but the `Unique` method of the datasets\n",
@@ -948,7 +948,7 @@
"\n",
"\n",
"# apply preprocessing function to input examples\n",
"datasets = datasets.map(preprocess_function, batched=True, load_from_cache_file=True)"
"dataset = dataset.map(preprocess_function, batched=True, load_from_cache_file=True)"
]
},
{
@@ -1091,8 +1091,8 @@
"trainer = Trainer(\n",
" model,\n",
" args,\n",
" train_dataset=datasets[\"train\"],\n",
" eval_dataset=datasets[\"test\"],\n",
" train_dataset=dataset[\"train\"],\n",
" eval_dataset=dataset[\"test\"],\n",
" data_collator=default_data_collator,\n",
" tokenizer=tokenizer,\n",
" compute_metrics=compute_metrics,\n",