Implementation and Operation

Question 1

ML model deployment sections

Answer 1

• interoperation with containers
• accelerating ML systems using Elastic Inference
• push model to edge using SageMaker Neo
• Security: IAM, VPC, KMS
• Right Instance Type
• A/B test in production environment

Question 2

Where are the models in SageMaker hosted?

Answer 2

in Docker containers registered in ECS

Question 3

How to distribute Tensorflow across multiple machines?

Answer 3

using a framework called

Horovod or Parameter Servers

Question 4

How to make containers compatible with SageMaker?

Answer 4

there is a library for that

run pip install sagemaker-containers

Question 5

How is the structure of a training container?

Answer 5

/opt/ml

• input
• model
• code
• output

input/config:
hyperparameters.json
resourceConfig.json

input/data:
/

code:
python or any other code that does the training should be here

output:
output goes here

output/failure:
failures goes here

model:
inference codes are here

Question 6

SageMaker on the edge

Answer 6

SageMaker Neo

ARM, Intel, Nvidia processors

avoid few hundred milliseconds of latency

Question 7

Codes that Neo compile

Answer 7

Tensorflow
MXNet
PyTorch
ONNX
XGBoost

Question 8

Neo comes with Compiler or Runtime?

Answer 8

comes with both

Question 9

How to pair Neo with IoT Greengrass?

Answer 9

take a Ne-compiled model and deploy to an https endpoint

• hosted on C5, M5, M4, P3, P2
• should be the same instance type used for compilation

in pairing with IoT Greengrass:

• train model on the cloud
• compile with Neo
• deploy to actual edge devices using IoT Greengrass

Greengrass uses Lambda functions

Question 10

Security in SageMaker

Answer 10

uses:

• IAM
• MFA
• SSL/TLS
• CloudTrail
• Encryption

be careful with PII

Question 11

How to keep data protected at rest in SageMaker

Answer 11

KMS

• jobs
• notebooks

anything under
/opt/ml
and /tmp
can be encrypted using KMS

Securing Training Data

• S3 encryptions
• also KMS

Question 12

Can you encrypt inter-node communications?

Answer 12

yes you can
it can increase the time and dollar with DL

also known as inter-container traffic encryption

Question 13

VPC with SageMaker

Answer 13

yes possible
also possible to cut internet from notebook

then need to set up vpc endpoint for s3

notebooks, training/inference containers are internet-enabled by default

Question 14

SageMaker logging and monitoring

Answer 14

CloudWatch can log, monitor and alarm on:

• invocation and latency of endpoints
• health of instance nodes
• Ground Truth (how much active workers are doing?)

CloudTrail records actions from users, roles, and services within SageMaker
- log files delivered to S3 for auditing

Question 15

SageMaker with spot instances, does that work?

Answer 15

it could be interrupted but you can use S3 checkpointing to pick up where you left off

it also can increase time

Question 16

Elastic Inference

Answer 16

to accelerates deep learning inferences

cheaper than a GPU instance

Question 17

How to do EI?

Answer 17

deploy the model into a CPU with a EI

e.g. ml.eia1.large

Question 18

can you do EI with notebooks?

Answer 18

yes you can

Question 19

Where EI function?

Answer 19

where it is deep learning things
pre-build TF or MXNet containers

image classification
object detection

Question 20

Can you do AS for inferences ?

Answer 20

yes you can

Question 21

SageMaker and AZ?

Answer 21

SageMaker distribute across multiple AZ for better resiliency

VPC with min 2 subnets 1/AZ