1_mme-hosting-stable-diffusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dab554c6",
   "metadata": {},
   "source": [
    "# Deploy Stable Diffusion on a SageMaker GPU Multi-Model Endpoint with Triton\n",
    "\n",
    "본 예제 코드는 AWS 공식 예제를 바탕으로 튜토리얼 코드를 추가하고 conda 가상 환경에서 사용자 정의 도커 컨테이너를 빌드하여 호스팅하는 방법으로 변경하였습니다.\n",
    "\n",
    "<br>\n",
    "\n",
    "## 1. Installs and imports \n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d9ee4791-0ab6-4a74-9a96-ed8fd00a3f82",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "import base64\n",
    "import jinja2\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "from IPython.display import display\n",
    "\n",
    "from utils.lib import download_model, get_triton_image_uri\n",
    "from pathlib import Path\n",
    "\n",
    "# variables\n",
    "s3_client = boto3.client(\"s3\")\n",
    "ts = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "# sagemaker variables\n",
    "role = get_execution_role()\n",
    "sagemaker_session = sagemaker.Session(boto_session=boto3.Session())\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "account_id = boto3.client(\"sts\").get_caller_identity().get(\"Account\")\n",
    "region = boto3.Session().region_name\n",
    "prefix = \"stable-diffusion-mme\"\n",
    "\n",
    "jinja_env = jinja2.Environment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fda567c-afaa-4a7e-a947-98f1b15cb358",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## 2. Save pretrained model\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d30d8b41-ffc3-49c6-831f-b70e08746284",
   "metadata": {},
   "source": [
    "`models` 디렉터리는 Triton Inference Server의 모델 리포지터리(model repository)를 의미합니다. 이 리포지토리의 각 하위 디렉토리는 해당 모델에 대한 설정 정보(`config.pbtxt`)와 모델(프레임워크 및 백엔드에 따라 모델 추론 코드나 모델 파라메터)을 아래와 같이 포함하고 있습니다.\n",
    "\n",
    "```\n",
    "<model-repository-path>/\n",
    "    <model-1-name>/\n",
    "        [config.pbtxt]\n",
    "        [<output-labels-file> ...]\n",
    "        <version>/\n",
    "            <model-definition-file>\n",
    "        <version>/\n",
    "            <model-definition-file>\n",
    "        ...\n",
    "    <model-2-name>/\n",
    "        [config.pbtxt]\n",
    "        [<output-labels-file> ...]\n",
    "        <version>/\n",
    "            <model-definition-file>\n",
    "        <version>/\n",
    "            <model-definition-file>\n",
    "        ...\n",
    "    ...\n",
    "```\n",
    "\n",
    "- Python: `model.py`\n",
    "- TorchScript: `model.pt`\n",
    "- TensorRT: `model.plan`\n",
    "- ONNX: `model.onnx`\n",
    "- TensorFlow: `model.graphdef`나 `model.savedmodel` 이름의 폴더\n",
    "\n",
    "\n",
    "본 예시에서는 파이썬 백엔드를 사용합니다.\n",
    "\n",
    "### config.pbtxt 개요\n",
    "backend, max_batch_size, input 및 output 등의 항목을 `config.pbtxt`에 작성합니다..\n",
    "\n",
    "- backend : 사용 프레임워크/백엔드 (예: \"tensorrt\", \"python\", \"onnxruntime\", \"tensorflow\", \"pytorch\")\n",
    "- platform : 모델 포맷 (예: \"tensorrt_plan\", \"tensorflow_savedmodel\", \"tensorflow_graphdef\")\n",
    "- max_batch_size : 클라이언트 혹은 Triton이 구성할 수 있는 dynamic batch size의 최댓값\n",
    "- input : 입력 tensor의 이름, data type 및 shape\n",
    "- output : 출력 tensor의 이름, data type 및 shape\n",
    "\n",
    "예시 \n",
    "\n",
    "```\n",
    "backend: \"onnxruntime\"\n",
    "max_batch_size: 8\n",
    "input [\n",
    "  {\n",
    "    name: \"input0\"\n",
    "    data_type: TYPE_FP32\n",
    "    dims: [ 224, 224, 3 ]\n",
    "  }\n",
    "]\n",
    "output [\n",
    "  {\n",
    "    name: \"output0\"\n",
    "    data_type: TYPE_FP32\n",
    "    dims: [ 1000 ]\n",
    "  }\n",
    "]\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0cd50657-e4e9-4532-9763-473d1a0d9d38",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models/sd_base/1/checkpoint\n",
      "Hugging Face model already exists! - [HF_MODEL_ID] stabilityai/stable-diffusion-2-1-base\n",
      "models/sd_depth/1/checkpoint\n",
      "Hugging Face model already exists! - [HF_MODEL_ID] stabilityai/stable-diffusion-2-depth\n",
      "models/sd_inpaint/1/checkpoint\n",
      "Hugging Face model already exists! - [HF_MODEL_ID] stabilityai/stable-diffusion-2-inpainting\n",
      "models/sd_upscale/1/checkpoint\n",
      "Hugging Face model already exists! - [HF_MODEL_ID] stabilityai/stable-diffusion-x4-upscaler\n"
     ]
    }
   ],
   "source": [
    "models_local_path = {\n",
    "    \"stabilityai/stable-diffusion-2-1-base\": \"models/sd_base/1/checkpoint\",\n",
    "    \"stabilityai/stable-diffusion-2-depth\": \"models/sd_depth/1/checkpoint\",\n",
    "    \"stabilityai/stable-diffusion-2-inpainting\": \"models/sd_inpaint/1/checkpoint\",\n",
    "    \"stabilityai/stable-diffusion-x4-upscaler\": \"models/sd_upscale/1/checkpoint\",\n",
    "}\n",
    "\n",
    "for model_name, model_local_path in models_local_path.items():\n",
    "    download_model(model_name, model_local_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95d58e0e",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## 3. Create Docker image and push it to ECR\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8973a7d2",
   "metadata": {},
   "source": [
    "파이썬 백엔드를 사용하는 경우 종속 패키지를 사용해야 하는 경우가 빈번하므로 conda pack을 사용한 자체 가상 환경이나 도커(docker) 이미지를 생성해야 합니다. AWS 공식 예제에서는 자체 가상 환경으로 모델을 배포하지만, 이 경우 단일 인스턴스 배포로만 제한되기에 오토스케일링을 적용할 수 없습니다. 따라서, 사용자 정의 도커 이미지를 생성해서 Amazon ECR(Elastic Container Registry)에 푸시하도록 하겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92df19fc-ea61-4fca-aa8a-bec2a4477d50",
   "metadata": {},
   "source": [
    "### Create Dockerfile\n",
    "\n",
    "사용자 정의 도커 이미지 생성 시, AWS에서 관리하는 도커 이미지를 베이스 이미지로 사용하는 것을 권장합니다. 아래 웹페이지를 참조하세요.\n",
    "- https://github.com/aws/deep-learning-containers/blob/master/available_images.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "de1f3668-ac27-428b-9bcb-191754140c1a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "triton_image_uri: 785573368785.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:23.05-py3\n"
     ]
    }
   ],
   "source": [
    "triton_image_uri, triton_account_id = get_triton_image_uri()\n",
    "print(f'triton_image_uri: {triton_image_uri}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d854e2cb-2dc6-4961-947a-10ec2293e99b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting docker/Dockerfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile docker/Dockerfile\n",
    "\n",
    "FROM {{triton_image_uri}}\n",
    "RUN pip install -U pip && \\\n",
    "    pip install --no-cache-dir sagemaker \\\n",
    "    boto3 \\\n",
    "    torch --extra-index-url https://download.pytorch.org/whl/cu118 \\\n",
    "    accelerate \\\n",
    "    transformers \\\n",
    "    diffusers \\\n",
    "    xformers\n",
    "\n",
    "ENV PYTHONUNBUFFERED=TRUE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4aa2bd22-e619-41ee-9adf-e8666c279ec1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1\t\u001b[34mFROM\u001b[39;49;00m\u001b[37m \u001b[39;49;00m\u001b[33m785573368785.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:23.05-py3\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "     2\t\u001b[34mRUN\u001b[39;49;00m\u001b[37m \u001b[39;49;00mpip\u001b[37m \u001b[39;49;00minstall\u001b[37m \u001b[39;49;00m-U\u001b[37m \u001b[39;49;00mpip\u001b[37m \u001b[39;49;00m&&\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     3\t\u001b[37m    \u001b[39;49;00mpip\u001b[37m \u001b[39;49;00minstall\u001b[37m \u001b[39;49;00m--no-cache-dir\u001b[37m \u001b[39;49;00msagemaker\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     4\t\u001b[37m    \u001b[39;49;00mboto3\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     5\t\u001b[37m    \u001b[39;49;00mtorch\u001b[37m \u001b[39;49;00m--extra-index-url\u001b[37m \u001b[39;49;00mhttps://download.pytorch.org/whl/cu118\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     6\t\u001b[37m    \u001b[39;49;00maccelerate\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     7\t\u001b[37m    \u001b[39;49;00mtransformers\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     8\t\u001b[37m    \u001b[39;49;00mdiffusers\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     9\t\u001b[37m    \u001b[39;49;00mxformers\u001b[37m\u001b[39;49;00m\n",
      "    10\t\u001b[37m\u001b[39;49;00m\n",
      "    11\t\u001b[34mENV\u001b[39;49;00m\u001b[37m \u001b[39;49;00m\u001b[31mPYTHONUNBUFFERED\u001b[39;49;00m=TRUE\u001b[37m\u001b[39;49;00m\n"
     ]
    }
   ],
   "source": [
    "template = jinja_env.from_string(Path(\"docker/Dockerfile\").open().read())\n",
    "Path(\"docker/Dockerfile\").open(\"w\").write(template.render(triton_image_uri=triton_image_uri))\n",
    "!pygmentize docker/Dockerfile | cat -n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fa2a412-5d5d-44e1-b138-4dc4635f5c2d",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Create script file for building and pushing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3272b935-1f57-42e6-836d-32ff7d05e645",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting build_and_push.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile build_and_push.sh\n",
    "#!/usr/bin/env bash\n",
    "\n",
    "# This script builds Docker container and pushes it to ECR, \n",
    "# so it is available for training and inference in Amazon Sagemaker.\n",
    "\n",
    "# Script takes 3 arguments:\n",
    "#    - image - required, this is container name which will be used when building locally and pushing to Amazon ECR;\n",
    "#    - tag - optional, if provided, it will be used as \":tag\" of your container name; otherwise, \":latest\" will be used;\n",
    "#    - dockerfile - optional, if provided, then docker will build container using specific dockerfile (e.g. \"Dockerfile.serving\"); otherwise, default \"Dockerfile\" will be used.\n",
    "\n",
    "# Usage examples:\n",
    "#    1. \"./build_and_push.sh d2-sm-coco-serving debug Dockerfile.serving\"\n",
    "#    2. \"./build_and_push.sh d2-sm-coco v2\"\n",
    "\n",
    "image=$1\n",
    "tag=$2\n",
    "dockerfile=$3\n",
    "\n",
    "if [ \"$image\" == \"\" ]\n",
    "then\n",
    "    echo \"Usage: $0 <image-name>\"\n",
    "    exit 1\n",
    "fi\n",
    "\n",
    "# Get the account number associated with the current IAM credentials\n",
    "account=$(aws sts get-caller-identity --query Account --output text)\n",
    "\n",
    "if [ $? -ne 0 ]\n",
    "then\n",
    "    exit 255\n",
    "fi\n",
    "\n",
    "# Get the region defined in the current configuration (default to us-east-2 if none defined)\n",
    "region=$(aws configure get region)\n",
    "\n",
    "if [ \"$tag\" == \"\" ]\n",
    "then\n",
    "    fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${image}:latest\"\n",
    "else\n",
    "    fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${image}:${tag}\"\n",
    "fi\n",
    "\n",
    "# If the repository doesn't exist in ECR, create it.\n",
    "aws ecr describe-repositories --repository-names \"${image}\" > /dev/null 2>&1\n",
    "\n",
    "if [ $? -ne 0 ]\n",
    "then\n",
    "    aws ecr create-repository --repository-name \"${image}\" > /dev/null\n",
    "fi\n",
    "\n",
    "# Get the login command from ECR and execute it directly (자신의 ECR 권한)\n",
    "$(aws ecr get-login --region ${region} --no-include-email)\n",
    "\n",
    "# Get public ECR access (해당 부분은 docker build 스크립트에서 사용하는 public ECR주소로 바꿔주어야 함)\n",
    "aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin {{triton_account_id}}.dkr.ecr.${region}.amazonaws.com\n",
    "\n",
    "if [ \"$dockerfile\" == \"\" ]\n",
    "then\n",
    "    docker build  -t ${image} .\n",
    "else\n",
    "    docker build -t ${image} . -f ${dockerfile}\n",
    "fi\n",
    "\n",
    "docker tag ${image} ${fullname}\n",
    "docker push ${fullname}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4d9f4583-e517-409e-8c0e-ce31cc32237f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "template = jinja_env.from_string(Path(\"build_and_push.sh\").open().read())\n",
    "Path(\"build_and_push.sh\").open(\"w\").write(template.render(triton_account_id=triton_account_id))\n",
    "#!pygmentize build_and_push.sh | cat -n\n",
    "!chmod +x build_and_push.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d371ba0-dbfc-450c-a1f5-b9d3526a0a61",
   "metadata": {},
   "source": [
    "### Build docker image and push it to ECR \n",
    "\n",
    "도커 이미지를 빌드하고 ECR에 푸시하는 데 몇 분의 시간이 소요됩니다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8a87589b-0ce0-4a14-95bd-f77661861fe9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "container = \"js-onboarding-mme\"\n",
    "tag = \"latest\"\n",
    "ecr_image_uri = f\"{account_id}.dkr.ecr.{region}.amazonaws.com/{container}:{tag}\"\n",
    "#! ./build_and_push.sh $container $tag docker/Dockerfile"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bfd05bb-a283-4548-95c5-d4740bb1b26f",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## (Optional) 4. Debugging in local development environment\n",
    "---\n",
    "\n",
    "SageMaker 엔드포인트를 배포하기 이전에 로컬 환경에서 테스트해볼 수 있습니다. \n",
    "\n",
    "- Triton 서버 구동: 아래 코드 셀에서 생성되는 `start_tritonserver.sh` 을 터미널에서 실행\n",
    "- Triton 클라이언트로 추론 수행: `2_simple-test-tritonclient.ipynb` 실행\n",
    "\n",
    "도커 컨테이너를 중단하고 이미지를 삭제하려면 `docker rm -f $(docker ps -qa)` 커맨드를 입력합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6682b1cc-5e3f-4a2b-8ec0-5432a176d409",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting start_tritonserver.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile start_tritonserver.sh\n",
    "docker run --gpus all --rm --shm-size=4G -p8000:8000 -p8001:8001 -p8002:8002 \\\n",
    "    -v {{root_dir}}/models:/models {{ecr_image_uri}} tritonserver \\\n",
    "    --model-repository=/models --model-control-mode=explicit --load-model=sd_base \\\n",
    "    --log-verbose=3 --log-info=1 --log-warning=1 --log-error=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "54ec1c61-7e6e-41d0-afb9-1f9fed7eb7d7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1\tdocker\u001b[37m \u001b[39;49;00mrun\u001b[37m \u001b[39;49;00m--gpus\u001b[37m \u001b[39;49;00mall\u001b[37m \u001b[39;49;00m--rm\u001b[37m \u001b[39;49;00m--shm-size=4G\u001b[37m \u001b[39;49;00m-p8000:8000\u001b[37m \u001b[39;49;00m-p8001:8001\u001b[37m \u001b[39;49;00m-p8002:8002\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     2\t\u001b[37m    \u001b[39;49;00m-v\u001b[37m \u001b[39;49;00m/home/ec2-user/SageMaker/llm-workshop/lab2-stable-diffusion/option3-triton-mme/models:/models\u001b[37m \u001b[39;49;00m\u001b[34m143656149352\u001b[39;49;00m.dkr.ecr.us-east-1.amazonaws.com/js-onboarding-mme:latest\u001b[37m \u001b[39;49;00mtritonserver\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     3\t\u001b[37m    \u001b[39;49;00m--model-repository=/models\u001b[37m \u001b[39;49;00m--model-control-mode=explicit\u001b[37m \u001b[39;49;00m--load-model=sd_base\u001b[37m \u001b[39;49;00m\u001b[33m\\\u001b[39;49;00m\n",
      "     4\t\u001b[37m    \u001b[39;49;00m--log-verbose=\u001b[34m3\u001b[39;49;00m\u001b[37m \u001b[39;49;00m--log-info=\u001b[34m1\u001b[39;49;00m\u001b[37m \u001b[39;49;00m--log-warning=\u001b[34m1\u001b[39;49;00m\u001b[37m \u001b[39;49;00m--log-error=\u001b[34m1\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n"
     ]
    }
   ],
   "source": [
    "template = jinja_env.from_string(Path(\"start_tritonserver.sh\").open().read())\n",
    "Path(\"start_tritonserver.sh\").open(\"w\").write(template.render(root_dir=os.getcwd(), ecr_image_uri=ecr_image_uri))\n",
    "!pygmentize start_tritonserver.sh | cat -n\n",
    "!chmod +x start_tritonserver.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88d0419b-eae2-4d9d-b275-fcd784e6c987",
   "metadata": {},
   "source": [
    "터미널(Terminal) 접속 후 `start_tritonserver.sh`를 실행하면 Triton 추론 서버 컨테이너가 실행됩니다. `2_simple-test-tritonclient.ipynb` 파일을 열어서 추른을 수행해 보세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ce8c882-c8cc-4135-9f6b-0f90707deee7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# %%bash\n",
    "# ./start_tritonserver.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89830242-247e-458a-b2c2-6637c6845872",
   "metadata": {
    "tags": []
   },
   "source": [
    "<br>\n",
    "\n",
    "## 4. Model upload to S3\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db38cac8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_root_path = Path(\"./models\")\n",
    "model_dirs = list(model_root_path.glob(\"*\"))\n",
    "model_upload_paths = {}\n",
    "\n",
    "for model_path in model_dirs:\n",
    "    model_name = model_path.name\n",
    "    tar_name = model_path.name + \".tar.gz\"\n",
    "    !tar -C $model_root_path -czvf $tar_name $model_name\n",
    "    model_upload_paths[model_name] = sagemaker_session.upload_data(path=tar_name, bucket=bucket, key_prefix=prefix)\n",
    "    !rm $tar_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1fcc106-eabc-4e61-b344-86a091db46d6",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## 5. SageMaker Endpoint (Local Mode)\n",
    "---\n",
    "\n",
    "로컬 모드는 필수로 수행할 필요는 없지만, 디버깅에 많은 도움이 됩니다. 다만, MME는 로컬 모드 구동 시 단일 모델만 로드되므로 `triton_env` 변수를 테스트하고자 하는 모델로 변경해 주시기 바립니다. (`\"SAGEMAKER_TRITON_DEFAULT_MODEL_NAME\": \"[YOUR-MODEL]\"`)\n",
    "\n",
    "또한, 로컬 모드 사용 시에는 모델을 S3에 반드시 업로드할 필요 없이 로컬 디렉터리에서도 로드할 수 있습니다. (`local_container` 변수 참조)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "817c4844-d19c-4e4b-9634-63df5cd41464",
   "metadata": {},
   "source": [
    "### MME configure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e328daa-3d5c-4c66-a6cc-7c80c4274dba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "triton_env = {\n",
    "    \"SAGEMAKER_TRITON_LOG_VERBOSE\": \"3\",\n",
    "    \"SAGEMAKER_TRITON_LOG_INFO\": \"1\",\n",
    "    \"SAGEMAKER_TRITON_LOG_WARNING\" : \"1\",\n",
    "    \"SAGEMAKER_TRITON_LOG_ERROR\" : \"1\",\n",
    "    \"SAGEMAKER_TRITON_DEFAULT_MODEL_NAME\": \"sd_base\"\n",
    "}\n",
    "\n",
    "model_data_url = f\"s3:https://{bucket}/{prefix}/\"  # s3 location where models are stored\n",
    "\n",
    "container = {\n",
    "    \"Image\": ecr_image_uri,\n",
    "    \"ModelDataUrl\": model_data_url,\n",
    "    \"Mode\": \"MultiModel\",\n",
    "    \"Environment\": triton_env\n",
    "}\n",
    "\n",
    "local_container = {\n",
    "    \"Image\": ecr_image_uri,\n",
    "    \"ModelDataUrl\": f\"{model_data_url}sd_base.tar.gz\",\n",
    "    \"Environment\": triton_env\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4786a402-4ab4-4c2e-a8b4-d77841415400",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set to True to enable SageMaker to run locally\n",
    "local_mode = True\n",
    "\n",
    "if local_mode:\n",
    "    from sagemaker.local import LocalSession\n",
    "    instance_type = \"local_gpu\"\n",
    "    sm_session = LocalSession()\n",
    "    sm_session.config = {'local': {'local_code': True}}\n",
    "    sm_client = sagemaker.local.LocalSagemakerClient()\n",
    "    smr_client = sagemaker.local.LocalSagemakerRuntimeClient()\n",
    "else:\n",
    "    instance_type = \"ml.g5.2xlarge\"\n",
    "    sm_session = sagemaker.Session()\n",
    "    sm_client = boto3.client(\"sagemaker\")\n",
    "    smr_client = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "instance_count = 1\n",
    "ts = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "sm_model_name = f\"{prefix}-mdl-{ts}\"\n",
    "endpoint_config_name = f\"{prefix}-epc-{ts}\"\n",
    "endpoint_name = f\"{prefix}-ep-{ts}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50369da7-dc74-460e-bf2e-006a8bab8f15",
   "metadata": {},
   "source": [
    "### Create Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a028477-dc2e-4841-95ab-cdd686a3542c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=sm_model_name, ExecutionRoleArn=role, PrimaryContainer=container\n",
    ")\n",
    "\n",
    "create_endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"InstanceType\": instance_type,\n",
    "            \"InitialVariantWeight\": 1,\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            \"ModelName\": sm_model_name,\n",
    "            \"VariantName\": \"AllTraffic\",\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "#print(\"Model Arn: \" + create_model_response[\"ModelArn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd057f4a-806b-46ca-8a78-66987417956c",
   "metadata": {
    "tags": []
   },
   "source": [
    "엔드포인트 생성까지 몇 분의 시간이 소요됩니다. 로컬 모드 사용 시에 도커 이미지가 저장되어 있다면, SageMaker 호스팅 인스턴스를 프로비저닝하고 도커 이미지를 다운로드하는 과정이 생략되므로 호스팅 환경보다 좀 더 빨리 배포됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bcbc6c1-7259-476a-8fff-d7639abc013a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65182c40-0498-4ef1-928d-02257fc21051",
   "metadata": {},
   "source": [
    "모델 서빙을 위한 도커 컨테이너가 구동되고 있음을 확인할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "259d0407-c138-4d0d-8810-5041d95498cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!docker ps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26eb51d6-9af2-4406-b06a-b844834650f6",
   "metadata": {},
   "source": [
    "### Inference test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0b1e511-309d-4af6-a2ba-e56dc55e998b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils.inference_lib import get_sample_binary, encode_image, decode_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed65bcef-5614-4d4e-b7c7-7c3432aac7f5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def invoke_endpoint(endpoint_name, payload, target_model):\n",
    "    response = smr_client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name,\n",
    "        Accept=\"application/json\",\n",
    "        ContentType=\"application/octet-stream\",\n",
    "        Body=json.dumps(payload),\n",
    "        TargetModel=target_model,\n",
    "    )\n",
    "    data = response[\"Body\"].read().decode(\"utf-8\")\n",
    "    output = json.loads(data)[\"outputs\"]\n",
    "    output_image = decode_image(output[0][\"data\"][0])\n",
    "    return output_image\n",
    "\n",
    "def invoke_endpoint_binary_json(endpoint_name, payload, target_model):\n",
    "    import re\n",
    "    request_body, header_length = get_sample_binary(payload)\n",
    "    response = smr_client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name,\n",
    "        ContentType=\"application/vnd.sagemaker-triton.binary+json;json-header-size={}\".format(\n",
    "            header_length\n",
    "        ),\n",
    "        Body=request_body,\n",
    "        TargetModel=target_model\n",
    "    )\n",
    "\n",
    "    data = response[\"Body\"].read()\n",
    "    ptn = re.compile(rb'\\{\"binary_data_size\":[0-9]*\\}')\n",
    "    match = json.loads(ptn.search(data).group().decode('utf-8'))\n",
    "    binary_data_size = match['binary_data_size']\n",
    "    binary_data = response_body[len(response_body)-binary_data_size:]\n",
    "    output_image = decode_image(binary_data)\n",
    "    return output_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdb36134-d401-449f-9e79-25977178045a",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = dict(\n",
    "    prompt=\"Infinity pool on top of a high rise overlooking Central Park\",\n",
    "    negative_prompt=\"blur, signature, low detail, low quality\",\n",
    "    gen_args=json.dumps(dict(num_inference_steps=50, guidance_scale=8)),\n",
    ")\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": name, \"shape\": [1, 1], \"datatype\": \"BYTES\", \"data\": [data]}\n",
    "        for name, data in inputs.items()\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f299ff3-3a84-437e-bc09-fd0d3a6285a9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "invoke_endpoint(endpoint_name, payload, target_model=\"sd_base.tar.gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e360b4e1-738e-407f-b9bd-fcc19b63d7ce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "invoke_endpoint_binary_json(endpoint_name, payload, target_model=\"sd_base.tar.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24a467c3-54b8-4199-a2f4-d1b0b0ac9b4f",
   "metadata": {},
   "source": [
    "### Delete Endpoint "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b71f0ff-e244-470c-8955-adf07b4398fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils.lib import delete_endpoint\n",
    "delete_endpoint(sm_client, endpoint_name)\n",
    "!sudo rm -rf /tmp/tmp*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb4c11a-4f8a-409e-832e-f27b8be07bb7",
   "metadata": {},
   "source": [
    "Create a SageMaker endpoint configuration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b078de8-b09e-4789-b7a2-84e2bfcab938",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## 6. SageMaker Endpoint (Hosting)\n",
    "---\n",
    "\n",
    "SageMaker 엔드포인트는 REST API를 통해 실시간 추론(real-time inference)을 수행하는 완전 관리형 서비스입니다. 기본적으로 분산 컨테이너로 고가용성, 다중 모델 로딩, A/B 테스트를 위한 인프라 환경(EC2, 로드밸런서, 오토스케일링, 모델 아티팩트 로딩 등)이 사전 구축되어 있기에 몇 줄의 코드만으로 엔드포인트가 자동으로 생성되기에, 모델을 프로덕션에 빠르게 배포할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af073c1d-49cd-4a84-9a4b-b4258a01c800",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set to True to enable SageMaker to run locally\n",
    "local_mode = False\n",
    "\n",
    "if local_mode:\n",
    "    from sagemaker.local import LocalSession\n",
    "    instance_type = \"local_gpu\"\n",
    "    sm_session = LocalSession()\n",
    "    sm_session.config = {'local': {'local_code': True}}\n",
    "    sm_client = sagemaker.local.LocalSagemakerClient()\n",
    "    smr_client = sagemaker.local.LocalSagemakerRuntimeClient()\n",
    "else:\n",
    "    instance_type = \"ml.g5.2xlarge\"\n",
    "    sm_session = sagemaker.Session()\n",
    "    sm_client = boto3.client(\"sagemaker\")\n",
    "    smr_client = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "instance_count = 1\n",
    "ts = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "sm_model_name = f\"{prefix}-mdl-{ts}\"\n",
    "endpoint_config_name = f\"{prefix}-epc-{ts}\"\n",
    "endpoint_name = f\"{prefix}-ep-{ts}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c058a84a-ff80-46cc-bfff-4fc9987697d2",
   "metadata": {},
   "source": [
    "### Create Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a016c915-73ba-46f8-9a87-89f1083e6eb6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=sm_model_name, ExecutionRoleArn=role, PrimaryContainer=container\n",
    ")\n",
    "\n",
    "create_endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"InstanceType\": instance_type,\n",
    "            \"InitialVariantWeight\": 1,\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            \"ModelName\": sm_model_name,\n",
    "            \"VariantName\": \"AllTraffic\",\n",
    "        }\n",
    "    ],\n",
    ")\n",
    "print(\"Model Arn: \" + create_model_response[\"ModelArn\"])\n",
    "print(\"Endpoint Config Arn: \" + create_endpoint_config_response[\"EndpointConfigArn\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f09a802-706e-49f4-ae36-cb4e32960b6f",
   "metadata": {},
   "source": [
    "엔드포인트를 생성하고 `InService` 상태로 전환될 때까지 기다립니다. 호스팅 인스턴스를 프로비저닝하고 호스팅 환경을 설정하는 데 몇 분의 시간이 소요됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39066acd-e31e-433d-9200-65f7bb3b6975",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_name = f\"{prefix}-ep-{ts}\"\n",
    "\n",
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "\n",
    "print(\"Endpoint Arn: \" + create_endpoint_response[\"EndpointArn\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e87e48cf-fbbe-421b-9ccd-97793bea5e6a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "def make_console_link(region, endpoint_name, task='[SageMaker LLM Serving]'):\n",
    "    endpoint_link = f'<b> {task} <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={region}#/endpoints/{endpoint_name}\">Check Endpoint Status</a></b>'   \n",
    "    return endpoint_link\n",
    "\n",
    "endpoint_link = make_console_link(region, endpoint_name)\n",
    "display(HTML(endpoint_link))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "898d7093-b9c8-45f0-92e3-49c6f31dec91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils.inference_lib import describe_endpoint\n",
    "describe_endpoint(endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abfee5c2-049d-4c96-89ea-725f99003bfe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(30)\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aff32048-bf34-4b27-abe7-580929bdbe39",
   "metadata": {},
   "source": [
    "### Query models \n",
    "\n",
    "MME는 최초 호출 시 S3에 저장된 모델을 호스팅 컨테이너로 복사하여 호스팅 메모리에 올리는 과정이 필수이기에, 콜트스타트가 존재합니다.\n",
    "최초 호출 이후에는 실시간 추론과 동등한 추론 성능을 보입니다.\n",
    "#### Stable Diffusion Base "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd2ca25e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "inputs = dict(\n",
    "    prompt=\"Infinity pool on top of a high rise overlooking Central Park\",\n",
    "    negative_prompt=\"blur, signature, low detail, low quality\",\n",
    "    gen_args=json.dumps(dict(num_inference_steps=50, guidance_scale=8)),\n",
    ")\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": name, \"shape\": [1, 1], \"datatype\": \"BYTES\", \"data\": [data]}\n",
    "        for name, data in inputs.items()\n",
    "    ]\n",
    "}\n",
    "\n",
    "# response = runtime_sm_client.invoke_endpoint(\n",
    "#     EndpointName=endpoint_name,\n",
    "#     ContentType=\"application/octet-stream\",\n",
    "#     Body=json.dumps(payload),\n",
    "#     TargetModel=\"sd_base.tar.gz\",\n",
    "# )\n",
    "# output = json.loads(response[\"Body\"].read().decode(\"utf8\"))[\"outputs\"]\n",
    "# original_image = decode_image(output[0][\"data\"][0])\n",
    "# original_image\n",
    "\n",
    "original_image = invoke_endpoint(endpoint_name, payload, target_model=\"sd_base.tar.gz\")\n",
    "print(\"Original image\")\n",
    "display(original_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4de5f93-1021-475d-ae7c-fb506fa7e458",
   "metadata": {},
   "source": [
    "#### Image Style Transfer\n",
    "Stable Diffusion base에서 생성한 이미지를 Depth 모델을 사용하여 다양한 이미지 스타일로 표현해 봅니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "517013a2-b6b1-4725-938c-de3a23f3fda2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "input_image = encode_image(original_image).decode(\"utf8\")\n",
    "\n",
    "inputs = dict(\n",
    "    prompt=\"highly detailed oil painting of an inifinity pool overlooking central park\",\n",
    "    image=input_image,\n",
    "    gen_args=json.dumps(dict(num_inference_steps=50, strength=0.8)),\n",
    ")\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": name, \"shape\": [1, 1], \"datatype\": \"BYTES\", \"data\": [data]}\n",
    "        for name, data in inputs.items()\n",
    "    ]\n",
    "}\n",
    "\n",
    "oil_painting = invoke_endpoint(endpoint_name, payload, target_model=\"sd_depth.tar.gz\")\n",
    "print(\"Oil painting\")\n",
    "display(oil_painting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d71af00d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "inputs = dict(\n",
    "    prompt=\"Infinity pool perched on a cliff overlooking Yellowstone National Park \",\n",
    "    image=input_image,\n",
    "    gen_args=json.dumps(dict(num_inference_steps=50, strength=0.8)),\n",
    ")\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": name, \"shape\": [1, 1], \"datatype\": \"BYTES\", \"data\": [data]}\n",
    "        for name, data in inputs.items()\n",
    "    ]\n",
    "}\n",
    "\n",
    "rocky_mountains = invoke_endpoint(endpoint_name, payload, target_model=\"sd_depth.tar.gz\")\n",
    "print(\"Yellowstone\")\n",
    "display(rocky_mountains)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3314b9e4-ebc2-49eb-8517-8da970123d4c",
   "metadata": {},
   "source": [
    "### Inpainting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d2aead9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "source_image = Image.open(\"sample_images/bertrand-gabioud.png\")\n",
    "\n",
    "image = encode_image(source_image).decode(\"utf8\")\n",
    "mask_image = encode_image(Image.open(\"sample_images/bertrand-gabioud-mask.png\")).decode(\"utf8\")\n",
    "inputs = dict(\n",
    "    prompt=\"building, facade, paint, windows\",\n",
    "    image=image,\n",
    "    mask_image=mask_image,\n",
    "    negative_prompt=\"tree, obstruction, sky, clouds\",\n",
    "    gen_args=json.dumps(dict(num_inference_steps=50, guidance_scale=10)),\n",
    ")\n",
    "\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": name, \"shape\": [1, 1], \"datatype\": \"BYTES\", \"data\": [data]}\n",
    "        for name, data in inputs.items()\n",
    "    ]\n",
    "}\n",
    "\n",
    "filled_image = invoke_endpoint(endpoint_name, payload, target_model=\"sd_inpaint.tar.gz\")\n",
    "\n",
    "print(\"source image\")\n",
    "display(source_image)\n",
    "\n",
    "print(\"filled image\")\n",
    "display(filled_image)\n",
    "#display(decode_image(output[0][\"data\"][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4f15f51-84dc-40d1-84f7-1cce850d80b2",
   "metadata": {},
   "source": [
    "#### Super resolution\n",
    "\n",
    "원본 출력 이미지의 크기를 512x512에서 128x128로 축소합니다. 그런 다음 Stable Diffusion 업스케일링 모델을 사용하여 이미지를 원래 512 해상도로 다시 업스케일링합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef41b693-5344-4ebc-a29b-1b931446a85a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "low_res_image = original_image.resize((128, 128))\n",
    "inputs = dict(\n",
    "    prompt=\"Infinity pool on top of a high rise overlooking Central Park\",\n",
    "    image=encode_image(low_res_image).decode(\"utf8\"),\n",
    ")\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": name, \"shape\": [1, 1], \"datatype\": \"BYTES\", \"data\": [data]}\n",
    "        for name, data in inputs.items()\n",
    "    ]\n",
    "}\n",
    "\n",
    "\n",
    "upscaled_image = invoke_endpoint(endpoint_name, payload, target_model=\"sd_upscale.tar.gz\")\n",
    "\n",
    "# response = runtime_sm_client.invoke_endpoint(\n",
    "#     EndpointName=endpoint_name,\n",
    "#     ContentType=\"application/octet-stream\",\n",
    "#     Body=json.dumps(payload),\n",
    "#     TargetModel=\"sd_upscale.tar.gz\",\n",
    "# )\n",
    "# output = json.loads(response[\"Body\"].read().decode(\"utf8\"))[\"outputs\"]\n",
    "#upscaled_image = decode_image(output[0][\"data\"][0])\n",
    "\n",
    "print(\"Low res image\")\n",
    "display(low_res_image.resize((512, 512)))\n",
    "\n",
    "print(\"Upscaled image\")\n",
    "display(upscaled_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6e7824e",
   "metadata": {},
   "source": [
    "## Clean up <a name=\"query\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f80a67ed-1b59-4040-b324-3d7427f08024",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils.lib import delete_endpoint\n",
    "delete_endpoint(sm_client, endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54a6ef61",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# delete models in respective paths\n",
    "for model_name, model_local_path in models_local_path.items():\n",
    "    print(model_local_path)\n",
    "    !rm -rf $model_local_path"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.m5.large",
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}