xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Ml ops (#77)

Browse Source 
* feat: new notebook

* feat: new notebook

* feat: get started notebook

* feat: get started notebook

* feat: refine notebook

* feat: refine notebook

* feat: add dataflow notebook

* feat: add dataflow notebook
This commit is contained in:
Andrew Ferlitsch
2021-09-27 13:25:22 -07:00
committed by GitHub
parent b4d533c38d
commit 3e80bc5479
3 changed files with 1456 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/community/ml_ops/stage1/get_started_bq_datasets.ipynb
+148 -11
View File
@@ -118,14 +118,18 @@
" - Preprocess the data in the dataframe.\n",
" - For large datasets:\n",
" - TensorFlow model training:\n",
" - Create a tf.data.Dataset generator from the BigQuery table for training TensorFlow models.\n",
" - Create a tf.data.Dataset generator from the BigQuery table.\n",
" - Specify the columns for the custrom training.\n",
" - Preprocess the data either:\n",
" - Within the generator (upstream)\n",
" - Within the model (downstream)\n",
" - XGBoost model training\n",
" - XGBoost model training:\n",
" - Use BigQuery ML builtin XGBoost training.\n",
" - Alternatively, create a DMatrix generator from CSV files extracted from BigQuery table.\n",
" - Pytorch model training:\n",
" - Extract the BigQuery to a pandas dataframe.\n",
" - Preprocess the data in the dataframe.\n",
" - Create a DataLoader generator from the pandas dataframe.\n",
"\n",
"\n",
"- Alternately:\n",
@@ -217,6 +221,26 @@
" ! pip3 install --upgrade tensorflow $USER_FLAG"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "install_xgboost"
},
"source": [
"Install the latest GA version of *XGBoost* library as well."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "install_xgboost"
},
"outputs": [],
"source": [
"! pip3 install -U xgboost $USER_FLAG"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -477,6 +501,28 @@
"from google.cloud import bigquery"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "import_xgboost"
},
"source": [
"#### Import XGBoost\n",
"\n",
"Import the XGBoost package into your Python environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "import_xgboost"
},
"outputs": [],
"source": [
"import xgboost as xgb"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -731,12 +777,7 @@
" ],\n",
")\n",
"\n",
"dataframe = rows.to_dataframe(\n",
" # Optionally, explicitly request to use the BigQuery Storage API. As of\n",
" # google-cloud-bigquery version 1.26.0 and above, the BigQuery Storage\n",
" # API is used by default.\n",
" create_bqstorage_client=True,\n",
")\n",
"dataframe = rows.to_dataframe()\n",
"print(dataframe.head())"
]
},
@@ -843,6 +884,104 @@
"print(tf_dataset.take(1))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataframe_to_bq"
},
"source": [
"### Create a BigQuery dataset from a pandas dataframe\n",
"\n",
"You can create a BigQuery dataset from a pandas dataframe using the BigQuery `create_dataset()` and `load_table_from_dataframe()` methods, as follows:\n",
"\n",
"- `create_dataset()`: Creates an empty BigQuery dataset, with the following parameters:\n",
" - `dataset_ref`: The `DatasetReference` created from the dataset_id -- e.g., samples.\n",
"- `load_table_from_dataframe()`: Loads one or more CSV files into a table within the corresponding dataset, with the following parameters:\n",
" - `dataframe`: The dataframe.\n",
" - `table`: The `TableReference` for the table.\n",
" - `job_config`: Specifications on how to load the dataframe data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dataframe_to_bq"
},
"outputs": [],
"source": [
"LOCATION = \"us\"\n",
"\n",
"SCHEMA = [\n",
" bigquery.SchemaField(\"station_number\", \"STRING\"),\n",
" bigquery.SchemaField(\"year\", \"INTEGER\"),\n",
" bigquery.SchemaField(\"month\", \"INTEGER\"),\n",
" bigquery.SchemaField(\"day\", \"INTEGER\"),\n",
" bigquery.SchemaField(\"mean_temp\", \"FLOAT\"),\n",
"]\n",
"\n",
"\n",
"DATASET_ID = \"samples\"\n",
"TABLE_ID = \"gsod\"\n",
"\n",
"\n",
"def create_bigquery_dataset(dataset_id):\n",
" dataset = bigquery.Dataset(\n",
" bigquery.dataset.DatasetReference(PROJECT_ID, dataset_id)\n",
" )\n",
" dataset.location = \"us\"\n",
"\n",
" try:\n",
" dataset = bqclient.create_dataset(dataset) # API request\n",
" return True\n",
" except Exception as err:\n",
" print(err)\n",
" if err.code != 409: # http_client.CONFLICT\n",
" raise\n",
" return False\n",
"\n",
"\n",
"def load_data_into_bigquery(dataframe, dataset_id, table_id):\n",
" create_bigquery_dataset(dataset_id)\n",
" dataset = bqclient.dataset(dataset_id)\n",
" table = dataset.table(table_id)\n",
"\n",
" job_config = bigquery.LoadJobConfig(\n",
" # Specify a (partial) schema. All columns are always written to the\n",
" # table. The schema is used to assist in data type definitions.\n",
" schema=[\n",
" bigquery.SchemaField(\n",
" \"station_number\", \"FLOAT\"\n",
" ), # <-- after one hot encoding\n",
" bigquery.SchemaField(\"year\", \"INTEGER\"),\n",
" bigquery.SchemaField(\"month\", \"INTEGER\"),\n",
" bigquery.SchemaField(\"day\", \"INTEGER\"),\n",
" bigquery.SchemaField(\"mean_temp\", \"FLOAT\"),\n",
" ],\n",
" # Optionally, set the write disposition. BigQuery appends loaded rows\n",
" # to an existing table by default, but with WRITE_TRUNCATE write\n",
" # disposition it replaces the table with the loaded data.\n",
" write_disposition=\"WRITE_TRUNCATE\",\n",
" )\n",
"\n",
" NEW_BQ_TABLE = f\"{PROJECT_ID}.{dataset_id}.{table_id}\"\n",
"\n",
" job = bqclient.load_table_from_dataframe(\n",
" dataframe, NEW_BQ_TABLE, job_config=job_config\n",
" ) # Make an API request.\n",
" job.result() # Wait for the job to complete.\n",
"\n",
" table = bqclient.get_table(NEW_BQ_TABLE) # Make an API request.\n",
" print(\n",
" \"Loaded {} rows and {} columns to {}\".format(\n",
" table.num_rows, len(table.schema), NEW_BQ_TABLE\n",
" )\n",
" )\n",
"\n",
"\n",
"load_data_into_bigquery(IMPORT_FILES, DATASET_ID, TABLE_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -955,7 +1094,7 @@
{
"cell_type": "markdown",
"metadata": {
"id": "bq_to_xgboost:gsod"
"id": "bq_to_xgboost"
},
"source": [
"### Read BigQuery table into XGboost DMatrix\n",
@@ -988,8 +1127,6 @@
},
"outputs": [],
"source": [
"import xgboost as xgb\n",
"\n",
"categories = dataframe[\"station_number\"].unique()\n",
"\n",
"one_hot = pd.get_dummies(categories)\n",
notebooks/community/ml_ops/stage1/get_started_dataflow.ipynb
+1278
View File
File diff suppressed because it is too large Load Diff
notebooks/community/ml_ops/stage1/get_started_vertex_datasets.ipynb
+30 -13
View File
@@ -123,7 +123,7 @@
" - image/video data:\n",
" - Export the data to a JSONL index file.\n",
" - Using the index file, convert the images/videos and labels to TFRecords.\n",
" - Create a tf.data.Dataset generator from the TFRrcords.\n",
" - Create a tf.data.Dataset generator from the TFRercords.\n",
" - text data:\n",
" - If text strings are embedded:\n",
" - Convert to CSV file.\n",
@@ -964,12 +964,7 @@
" ],\n",
")\n",
"\n",
"dataframe = rows.to_dataframe(\n",
" # Optionally, explicitly request to use the BigQuery Storage API. As of\n",
" # google-cloud-bigquery version 1.26.0 and above, the BigQuery Storage\n",
" # API is used by default.\n",
" create_bqstorage_client=True,\n",
")\n",
"dataframe = rows.to_dataframe()\n",
"print(dataframe.head())"
]
},
@@ -1102,12 +1097,7 @@
" ],\n",
")\n",
"\n",
"dataframe = rows.to_dataframe(\n",
" # Optionally, explicitly request to use the BigQuery Storage API. As of\n",
" # google-cloud-bigquery version 1.26.0 and above, the BigQuery Storage\n",
" # API is used by default.\n",
" create_bqstorage_client=True,\n",
")\n",
"dataframe = rows.to_dataframe()\n",
"\n",
"new_stats = tfdv.generate_statistics_from_dataframe(\n",
" dataframe=dataframe,\n",
@@ -1146,6 +1136,33 @@
"print(feature_spec)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "text_dataset_to_pandas"
},
"source": [
"### Export `Text Dataset` to pandas dataframe.\n",
"\n",
"The property `gca_resource.metadata['inputConfig']['gcsSource']['uri']` contains the list of the one or more imported CSV files.\n",
"\n",
"To create a dataframe from multiple CSV sources, you read each CSV file and concatenate the dataframes together."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "text_dataset_to_pandas"
},
"outputs": [],
"source": [
"all_files = dataset.gca_resource.metadata[\"inputConfig\"][\"gcsSource\"][\"uri\"]\n",
"df = pd.concat(pd.read_csv(f) for f in all_files)\n",
"\n",
"print(df.head)"
]
},
{
"cell_type": "markdown",
"metadata": {