Add documentation for building chat applications using LLaMA 2. (#2494)

notebooks/community/model_garden/model_garden_pytorch_llama2_peft.ipynb

@@ -635,6 +635,8 @@
         "id": "aUMsewPDj_pS"
       },
       "source": [
+        "### Moderate model predictions\n",
+        "\n",
         "Text moderation analyzes a document against a list of safety attributes, which include \"harmful categories\" and topics that may be considered sensitive."
       ]
     },
@@ -653,6 +655,76 @@
         "    show_text_moderation(generated_text, response)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "f615c03d6638"
+      },
+      "source": [
+        "### Build chat applications with LLaMA2\n",
+        "\n",
+        "Following the [prompt template recognized by LLaMA2 models](https://huggingface.co/blog/llama2#how-to-prompt-llama-2), you can build multi-turn chat applications while having control over the system prompt.\n",
+        "\n",
+        "The output of the Vertex endpoint is `Prompt:\\n{{ model_prompt }}\\nOutput:\\n{{ model_output }}`. You can format this output based on the prompt template to build chat applications in the following way:\n",
+        "\n",
+        "In the first turn of the conversation, the user of the chat application sends `user_message_1`. With `user_message_1`, the chat application can formulate `model_prompt` as the following:\n",
+        "```\n",
+        "<s>[INST] <<SYS>>\n",
+        "{{ system_prompt }}\n",
+        "<</SYS>>\n",
+        "\n",
+        "{{ user_message }} [/INST]\n",
+        "```\n",
+        "The chat application then sends `model_prompt` to the Vertex endpoint and receives an output of the form: `Prompt:\\n{{ model_prompt }}\\nOutput:\\n{{ model_output }}`. The corresponding endpoint output contains both the original `model_prompt` as well as the generated `model_output`. The chat application can then exact `model_output` to present to the user of the chat application.\n",
+        "\n",
+        "In the second turn of the conversation, the user of the chat application sends `user_message_2` and the chat application can formulate `model_prompt` as:\n",
+        "```\n",
+        "<s>[INST] <<SYS>>\n",
+        "{{ system_prompt }}\n",
+        "<</SYS>>\n",
+        "\n",
+        "{{ user_message_1 }} [/INST] {{ model_answer_1 }} </s><s>[INST] {{ user_message_2 }} [/INST]\n",
+        "```\n",
+        "where `model_answer_1` is defined to be `model_output` from the previous turn. Again, the Vertex endpoint will generate an output of the form: `Prompt:\\n{{ model_prompt }}\\nOutput:\\n{{ model_output }}`, and the chat application can present `model_output` to the user.\n",
+        "\n",
+        "The same approach applies to further turns of the conversation, where all prior user messages and model answers are concatenated in the prompt.\n",
+        "\n",
+        "Below we show how [the HuggingFace example](https://huggingface.co/blog/llama2#how-to-prompt-llama-2) can be used with Vertex endpoints. Assume that in the first turn of the conversation, the chat application receives the user message: \"There's a llama in my garden 😱 What should I do?\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "749d516ea1db"
+      },
+      "outputs": [],
+      "source": [
+        "instances = [\n",
+        "    {\n",
+        "        \"prompt\": \"\"\"<s>[INST] <<SYS>>\n",
+        "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n",
+        "\n",
+        "If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n",
+        "<</SYS>>\n",
+        "\n",
+        "There's a llama in my garden 😱 What should I do? [/INST]\"\"\",\n",
+        "        \"max_tokens\": 50,\n",
+        "        \"temperature\": 1.0,\n",
+        "        \"top_p\": 1.0,\n",
+        "        \"top_k\": 10,\n",
+        "    },\n",
+        "]\n",
+        "response = endpoint_without_peft.predict(instances=instances)\n",
+        "\n",
+        "endpoint_output = response.predictions[0]:\n",
+        "print(f\"[Endpoint output]\\n{endpoint_output}\\n\")\n",
+        "\n",
+        "model_output_start_index = endpoint_output.find(\"\\nOutput:\\n\") + len(\"\\nOutput:\\n\")\n",
+        "model_output = endpoint_output[model_output_start_index:]\n",
+        "print(f\"[Model output (model answer) to show the user]\\n{model_output}\")"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {