More Solution Architectures + HPC Flashcards
What services can an event processing architecture involve?
SQS + Lambda
SNS + Lambda
SDK (Dev tools) + SNS + multiple SQS (Fan out pattern).
S3 Events to SQS
S3 Events to Lambda
S3 Events to SNS
S3 Events with EventBridge is by default, from Eventbridge you can send events to multiple services
S3 events are notifications of when a object is uploaded for example.
For any AWS API Call:
CloudTrail + EventBridge + SNS alert + Any Subscriber.
Events external to AWS:
API Gateway to kinesis data stream + kinesis data firehose + s3.
What services can a caching architecture involve?
Dynamic content caching:
Cloudfront + API Gateway + EC2 App + RDS + Redis or Memcached or DAX.
Here cloudfront, api gw, and redis or mem or dax have caching for different purposes.
Static content caching: Cloudfront edge + S3
What is High performance computing?
HPC is when you use a very high amount of resources in a short time.
The cloud is perfect for HPC. You can create a very high number of resources in no time, and then destroy them when you want to.
You can speed up the time it takes to get results by adding more resources.
You only pay for what you use.
What are HPC use cases?
perform genomics, computational chemestry, finantial risk modeling, ML, weather prediction, deep learning, etc.
What AWS services and features can help you perform high performance computing?
For data transfer:
-AWS Direct Connect.
-Snowball.
-Datasync.
For high end computing:
-CPU optimized or GPU optimized instances.
-Spot instances or spot fleets for cheap horizontal scaling and temporary compute power.
-EC2 placement group CLUSTER type for high performance and low latency between your ec2 instance fleet.
Compute and Networking:
-EC2 enhanced networking (SR-IOV): This gives you higher bandwidth, higher PPS, packets per second, and lower network latency.
-ENA: Elastic Network Adapter: Gives you a Network speed of up to 100Gbps.
-EFA: Elastic Fabric Adapter: An improved ENA, only for linux. Great for inter node communication, and tightly coupled workloads (works well with cluster placement groups).
Data Storage:
Instance attached storage:
EBS: Scales up to 256000 IOPS with IO2
Instance Store: Scales up to millios of IOPS, linked to the EC2 instance, lowest latency.
Network Storage:
S3: Large object storage.
EFS: Scale IOPS or Provision IOPS.
FSx for Lustre is a file system dedicated to HPC. Millions of IOPS and backed by S3 in the backend.
Automation and Orchestration:
AWS Batch to run batch concurrent jobs, that deploy hundreds of EC2 instances in batches.
AWS Parallel Cluster: A Cluster management tool to deploy HPC in AWS. It automtes creation of VPC, Subnets, Cluster type and instance types. It has the ability to enable EFA, so you can use EFA in conjunction with Parallel Cluster, for improved network performance.
Whats the difference between an ENI, and ENA, and an ENF?
ENI is the standard network card in aws.
ENA is Elastic Network Adapter: Gives you a Network speed of up to 100Gbps.
EFA: Elastic Fabric Adapter: An improved ENA, only for linux. Great for inter node communication, and tightly coupled workloads. It leverages the Message Passing Interface (MPI) standard. This bypasses the underlying linux OS to provide lower latency, more reliable transport.
What is AWS Parallel Cluster?
AWS Parallel Cluster: A Cluster management tool to deploy HPC in AWS. It automtes creation of VPC, Subnets, Cluster type and instance types. It has the ability to enable EFA, so you can use EFA in conjunction with Parallel Cluster, for improved network performance.
What is AWS Batch?
AWS Batch to run batch concurrent jobs, that deploy hundreds of EC2 instances in batches.
Which services or features can help you with Automation and Orchestration when performing high performance computing (HPC) in AWS?
AWS Batch and Parallel Cluster
Which services of features can help you with storing your data when performing high performance computing (HPC) in AWS?
Data Storage:
Instance attached storage:
EBS: Scales up to 256000 IOPS with IO2
Instance Store: Scales up to millios of IOPS, linked to the EC2 instance, lowest latency.
Network Storage:
S3: Large object storage.
EFS: Scale IOPS or Provision IOPS.
FSx for Lustre is a file system dedicated to HPC. Millions of IOPS and backed by S3 in the backend.
Which services or features can help you with improving compute and networking capabilities when performing high performance computing (HPC) in AWS?
Compute and Networking:
-EC2 enhanced networking (SR-IOV): This gives you higher bandwidth, higher PPS (packets per second), and lower network latency.
-ENA: Elastic Network Adapter: Gives you a Network speed of up to 100Gbps.
-EFA: Elastic Fabric Adapter: An improved ENA, only for linux. Great for inter node communication, and tightly coupled workloads (works well with cluster placement groups).
Which services or features can help you with transfering data into AWS for performing high performance computing (HPC) in AWS?
For data transfer:
-AWS Direct Connect.
-Snowball.
-Datasync.
Which services or features can help you getting the computing power you need when performing high performance computing (HPC) in AWS?
For high end computing:
-CPU optimized or GPU optimized instances.
-Spot instances or spot fleets for cheap horizontal scaling.
-EC2 placement group CLUSTER type for high performance and low latency between your ec2 instance fleet.
Which file system is dedicated to performing HPC?
FSx for Lustre. You can achieve Millions of IOPS with it.
What can you use in AWS yo make an EC2 instance highly available?
Elastic ip + EC2 instance + Standby instance + lambda + CloudWatch to detect failures+ Lambda for failover to the standby instance in case of failure.
Elastic ip + EC2 instance + ASG with desired capacity of 1 (launched instances have user data to attach them to the elastic ip): In this case if the instance is terminated, the ASG will launch a new instance in another AZ and it will be attached the elastic ip.
Same example as previous, but with persistent data in an EBS Volume: ASG Lifecycle hook on instance termination to snapshot the instance EBS Volume.
Another ASG Lifecycle hook on instance launch to create EBS Volume out of the EBS Snapshot into the correct AZ. and then attach it to the new EC2 instance.
What are ASG Lifecycle hooks?
This ASG feature allows you to perform custom actions when an instance state change is detected. For example make a snapshot of an EBS Volume.
You are working on a Serverless application where you want to process objects uploaded to an S3 bucket. You have configured S3 Events on your S3 bucket to invoke a Lambda function every time an object has been uploaded. You want to ensure that events that can’t be processed are sent to a Dead Letter Queue (DLQ) for further processing. Which AWS service should you use to set up the DLQ?
Lambda.
The Lambda function’s invocation is “asynchronous”, so the DLQ has to be set on the Lambda function side.
(When you invoke a function, you can choose to invoke it synchronously or asynchronously. With synchronous invocation, the service that called the function has to wait for the function to process the event and return a response to it. With asynchronous invocation, Lambda queues the event for processing and the calling service doesn’t have to get a response from lambda. Actually s3 can’t process responses from lambda at all, so it always uses asyncronous invocation)
Lambda sets ups the events to be sent to the SQS DLQ. S3 can’t.
You could also have set up the s3 event with eventbridge and through eventbridge send the event to an SQS queue, then process it with lambda, and then SQS can handle the DLQ if lambda fails to process it. But the question specifies you are using s3 + lambda.
In this case it would be SQS who handles the retries and the DLQ, not lambda.
As a Solutions Architect, you have created an architecture for a company that includes the following AWS services: CloudFront, Web Application Firewall (AWS WAF), AWS Shield, Application Load Balancer, and EC2 instances managed by an Auto Scaling Group. Sometimes the company receives malicious requests and wants to block these IP addresses. According to your architecture, Where should you do it?
WAF.
Only cloudfront sees the incoming ips from the clients.
Shield if for DDoS protection, not a firewall.
You have a set of Linux EC2 instances deployed in a Cluster Placement Group in order to perform High-Performance Computing (HPC). You would like to maximize network performance between your EC2 instances. What should you use?
EFA: Elastic Fabric Adapter
