HPC on AWS Flashcards
Data Transfer - HPC
Snowball, snowmobile (terabytes/petabytes worth of data
AWSDataSync to store on s3,EFS,FSx for Windows etc
Direct Connect - dedicated private line from your data center to AWS. reducce your network costs, increase bandwidth throughput, and provide a more consistent network experience than internet connections
Compute & Networking - HPC
EC2 instances that are cPU or GPU optimized
EC2 fleets ( spot instances or spot fleets)
Placement Groups ( Cluster placement groups)
Enhanced Networking
Elastic Network Adapters - higher i/o lower CPU.
It usses single root I/O virtualization (SR-IOV) to provide high performance networking capabilities .. higher i/o lower CPU. No additional charge for enhanced networking.
Elastic Fabric Adapters - Lower or more consistency and higher throughput.
OS-Bypass
is a network device that you attach to your ec2 instance to accelerate HPC and machine learning applications.
Storage services that allow HPC
Instance attached storage:
EBS: scales up to 64,000 iops with provisioned iops.
instance store : scales up to millions of iops; low latency.
Network storage:
Amazon s3: Distributed object based storage not a file system.
Amazon EFS: Scales iops based on total size or use provisioned iops
Amazon Fsx for lustre = HPC optimized distributed file system; millions of iops; which is also backed by s3.
Orchestration and Automation services
AWS Batch :
which lets you run thousands of batch computing jobs.
allows to run a single job which spans in multiple ec2 instances
you can easily schedule jobs and launch Ec2 instances according to your needs
AWS Parallel Cluster:
Open source cluster management tool that makes it easy for you to deploy and manage hPC clusters on AWS
PARALLEL CLUSTERS USES A SIMPLE FILE TO MODEL AND PROVISION ALL THE RESOURCES NEEDED.
AUTOMATES CREATION OF VPC, SUBNET, CLUSTER TYPES AND INSTACNE STYPES