The article is here to help you with the above. We are going to show how you can easily deploy custom ML models using Docker, FastAPI, and AWS Sagemaker in 9 simple steps:<\/strong><\/p>\n\n\n\n For the sake of simplicity, we are going to use the pre-trained HuggingFace model, a fine-tuned BERT model that is ready to use for Named Entity Recognition. It recognizes four types of entities: Locations (LOC), Organisations (ORG), Persons (PER), and Miscellaneous (MISC). In reality, this could be any model you\u2019d like to deploy.<\/p>\n\n\n\n First of all, you need to create a specific directory and file structure (see below) within your working directory.<\/p>\n\n\n\n Directory opt\/ <\/strong>will be your root. The main.py<\/em> should contain the inference code, that reads the model from a specific path: ml\/model <\/strong>and exposes REST API for performing predictions \u2014 we are using FastAPI which is a high-performance web framework for developing APIs with Python. Let\u2019s take a look at the script, especially the invocations <\/em>and ping<\/em> entrypoints that are required by AWS SageMaker.<\/p>\n\n\n Then we have Dockerfile. It executes several command-line instructions step by step and creates a container based on the indicated image and requirements.<\/p>\n\n\n Requirements.txt indicates what packages are required and to be installed while creating the docker image.<\/p>\n\n\n Finally, there is run_server.sh. It <\/em>starts the machine learning service hosting your model and inference code. The host parameter specifies the IP address the server will listen to, and the port parameter specifies the port number. Don\u2019t modify them, as they come as required by SageMaker.<\/p>\n\n\n Go to the created repository and click on the View push commands<\/em> <\/strong>button to view the steps to push an image to your new repository. Make sure you are in the location of the If all goes well, you should be able to see your ECR image on AWS.<\/p>\n\n\n\n Models must be packaged as compressed tar files ( To compress the \n
Train the model locally or in a cloud-based environment<\/strong><\/h2>\n\n\n\n
Dockerize the inference code<\/strong><\/h2>\n\n\n\n
![\"Structure](\"https:\/\/sii.ua\/blog\/wp-content\/uploads\/2024\/01\/1-10-1.png\")
<\/a>\n# main.py\nfrom fastapi import FastAPI, Request\nfrom transformers import pipeline\n\nMODELS_PATH = "ml\/model\/"\n\napp = FastAPI()\n\n\n@app.get('\/ping')\nasync def ping():\n return {"message": "ok"}\n\n@app.on_event('startup')\ndef load_model():\n classifier = pipeline("ner", model=MODElS_PATHA)\n logging.info("Model loaded.")\n return classifier\n\n@app.post('\/invocations')\ndef invocations(request: Request):\n json_payload = await request.json()\n\n inputs = [records["scope"] for records in json_payload]\n output = [{"prediction": classifier(input)} for input in inputs ]\n return output\n<\/pre><\/div>\n\n\n\n
\n# Dockerfile\nFROM python:3.8-slim\n\n# copy the code and requirements\nCOPY main.py \/opt\/\nCOPY requirements.txt \/opt\/\nCOPY run_server.sh \/opt\/serve\n\nWORKDIR \/opt\n\n# install required python stuff\nRUN pip install -r requirements.txt\n\n# setup the executable path\nENV PATH="\/opt\/:${PATH}"\n<\/pre><\/div>\n\n\n\n# requirements.txt\nfastapi==0.89.1\nuvicorn==0.20.0\ntransformers==4.25.1\n<\/pre><\/div>\n\n\n
\n# run_server.sh\nuvicorn main:app --proxy-headers --host 0.0.0.0 --port 8080\n<\/pre><\/div>\n\n\n
Push the Docker image to Amazon Elastic Container Registry (ECR)<\/h2>\n\n\n\n
\n
\n
<\/a>.Dockerfile<\/code> and run the commands one by one.<\/p>\n\n\n\naws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin XXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com\ndocker build -t your_repository_name .\ndocker tag your_repository_name:latest XXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com\/your_repository_name:latest\ndocker push XXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com\/your_repository_name:latest\n<\/pre><\/div>\n\n\n
Save model artifacts on S3 bucket<\/strong><\/h2>\n\n\n\n
*.tar.gz<\/code>) and saved on the S3 bucket. Let\u2019s say your model\u2019s directory looks like this:<\/p>\n\n\n\n<\/a>model<\/code> file make sure that you are in ml\/<\/strong> location and run the following commands:<\/p>\n\n\n