Architect Certification Flashcards
Recovery Time Objective (RTO)
Maximum amount of time in which a service can remain unavailable before it is classed as damaging to the business
Recovery Point Objective (RPO)
Maximum amount of time for which data could be lost for a service
Ways of getting data in/out of AWS from on-premise
Direct Connect, VPN Connection, Internet Connection
How much data does a Snowball appliance hold
50-80 TB
How much data does a Snowmobile hold
100 PB
Storage Gateway
Connects on-premise database to AWS S3
S3 classes
Standard Class (Durability = 11 9’s, Availability = 4 9’s), Infrequence Access (IA) (Durability = 11 9’s, Availability = 3 9’s), Amazon Glacier (Duribility = 11 9’s, Availabiliy = N/A). IA is often used for backup data. Gracias is used for “cold storage”. Standard is most expensive. Glacier is least expensive.
AWS Artifact
Allows access to AWS Compliance Reports which are useful to auditors. Reports include the scope (AWS services, regions, etc.)
S3 capacity
Files from 1 byte to 5 TB (later lesson says 0 byes to 5TB)
S3 Class: Standard
Automatically replicates data across AZs within a region. Can encrypt data in transit and at rest. Has data management capabilities so that data can be moved to other S3 classes or deleted for cost optimization.
S3 Class: Infrequent Access (IA)
Only difference from standard class is lower cost and lower availability.
S3 Class: Amazon Glacier
Stores data in archives instead of buckets. Archives can save up to 40TB. Archives are stored within vaults.
AMI
Baseline EC2. Can be purchased through Marketplace or selected from community versions.
Instance Type
The size of an instance based on several parameters. Key parameters are vCPUs, memory, instance storage and network performance. Instances are grouped into families.
Instance Families
Micro (low throughput services), General Purpose (small to medium databases, test servers and backend servers), Compute optimized (compute intensive, video processing, scientific apps), GPU (graphics intensive apps), FPGA (massively parallel such as genomics and financial computing), Memory Optimized (real-time in-memory apps), Storage Optimized (uses SSD to reduce latency for very high I/O like noSQL databases)
Instance Purchase Options
On-Demand, Reserved, Scheduled, Spot, On-Demand Capacity Reservations
On Demand Instances
Launch at any time, can be used for as long as you want, flat rate, typically used for short term uses, best fit for testing and development
Reserved Instances
Purchase is made for 1-3 year term in exchange for a discount. Instances are either paid for all upfront, partial upfront or no upfront.
Scheduled Instances
Used for daily, weekly or monthly tasks.
Spot Instances
Must big on available EC2 resources. As long as the bid price is above the fluctuating price set by Amazon, get to use the instance. If the big falls below the price, a 2 minute warning is issued before termination. Only useful for processing that can suddenly interrupted.
On Demand Capacity Reservations
Reserve capacity based on instance type, platform and tenancy within a particular AZ for any length of time. Can be used in conjunction with reserve instance discounts.
Shared Tenancy
EC2 will run on any available host regardless of who else is running on that same server.
Dedicates Instances
EC2 runs on dedicated hardware.
Dedicated Host
Similar to dedicated instances but allows the same host to be used by multiple instances. Also allowed for running licensed software.
User Data
Allows you run commands upon the first boot to install software or apply software patches.
Persistent Storage
Attaching EBS volumes. Network attached. Snapshots and backups can be created. Can be encrypted.
Ephemeral Storage
Storage on the EC2 instance. Data s lost as soon as EC2 is stopped for terminated, but will remain if rebooted.
Secuity Group
An instance level firewall to control ingress and egress.
Key Pair
Made up of a public key and private key. Function is to encrypt the login information for Linux and Windows EC2 instances, and then decrypt the same information allowing you to authenticate onto the instance. Allows you to logon to Linux via SSH. The public key is held by AWS and the private key is your responsibility and must not get lost. The key pair can be used for multiple instances. After initial login, you can setup less privileged access controls.
Security - OS patches
It is your responsibility to download and install OS patches.
System Status Checks
Checks AWS components out of our control. If there is an issue, the best thing to do is stop and start the instance which would cause the instance to start on another host resolving the problem. Don’t reboot because it will cause the instance to continue running on the same physical server.
Instance Status Checks
If this fails, your input will be required to resolve. It looks at the EC2 instance itself instead of looking at the host.
EC2 Container Service
Allows you to run docker-enabled applications packaged as containers across a cluster of EC2 instances. The burden of managing the cluster is the responsibility of AWS specifically AWS Fargate. There is no need to install management or monitoring software.
Way of launching an ECS cluster
Fargate launch or EC2 launch
Fargate Launch
Requires you to specify CPU and memory and define networking and IAM policies in addition to having to package application in containers.
ECS - EC2 Launch
You are responsible for patching and scaling your instances. You can specify instance type and how many instances in a cluster.
Monitoring containers
Done through Cloudwatch.
ECS Cluster
Collection of EC2 instances. Security Groups, Load Balancing and Auto Scaling can be used. Instance operates in much the same way as a single EC2 instance. Clusters act as a resource pool, aggregating resources such as memory and CPU. Dynamically scalable. Can only scale in a single region bus multiple AZs. Containers can be scheduled to deploy across cluster. Instances within the cluster also have docker daemon and an ECS agent.
ECR
Elastic Container Registry. Provides a secure location to store and manager docker images.
ECR components
Registry, Authorization Token, Repository, Repository Policy, Image
ECR Registry
Allows you to host and store docker images as well as create image repositories. Access can be controlled by IAM policies as well as repository policies. Before the docker client can access, it needs to be authenticated as an AWS user via an Authorization token.
ECR authorization token
Run the CLI get-login command which will generate an output response which will be your docker login command. This will produce an authorization token that can be used within the registry for 12 hours.
ECR Repositories
Allow you to group together and secure different docker images. Can create multiple repositories so docker images can be organized into different categories. Using policies from IAM and repository policies you can assign set permissions to each repository.
ECR Repository Policy
Resource based policies. Need to ensure you add a principle to the policy to determine who has access and what permissions they have. AWS user will need access to the ecr:GetAuthorizationToken API call.
ECR images
Can be pushed and pulled using docker commands once the security has been configured.
ECS for Kubernetes (EKS)
Allows you to run Kubernetes in AWS without having provision or manager the control plane. You just need to provide and maintain the worker nodes.
Kubernetes control plane
Schedules containers onto the nodes, tracks the state of all Kubernetes objects. AWS responsible for provisioning, scaling and manages this across different AZs.
Worker Node
Worker machine in Kubernetes. Runs as an on-demand EC2 instance and contains software to run containers. A specific API is used. Once provisioned, they can connect to EKS using an endpoint.
Steps to run EKS
(1) IAM service role that must be created to allow EKS to provision and config specific resources Needs to have the following permission policies attached: AmazonEKSServicePolicy, AmazonEKSClusterPolicy (2) Cloudformation stack must be created and run for you to run with EKS. (3) Install kubectl and AWS-IAM-Authenticator (4) Use the EKS console to create the EKS cluster. (5) Configure kubectl: Use the update-kubeconfig command via the AWS CLI to create a kubeconfig file. (6) Provision and configure worker nodes. (7) Configure worker node to join EKS cluster. — Your cluster and worker nodes are now ready for you to deploy your applications.
Elastic Beanstalk
Takes you web application code and automatically provisions and deploys the required resources to make it operational.
Application Version
Very specific reference to a section of deployable code. and will typically point to S3.
Environment
Refers to an application environment that has been deployed onto AWS resources which are configured and provisioned by Elastic Beanstalk. The environment in comprised of all the resources created by Elastic Beanstalk and not just the EC2 instance.
Environment Configurations
Collection of parameters and settings that dictate how the environment will have its resources provisioned.
Environment Tier
If it handles HTTP requests, it will run in a web server environment. If it does not process HTTP requests but processes messages from SQS, it will run in a worker environment.
Configuration Template
Baseline for creating a new unique environment configuration.
Platform
Culmination of components in which you can build your components using Elastic Beanstalk. OS of the instance, the programming language, server type (web or application) and components of Elastic Beanstalk itself.
Applications
An application is a collection of different elements such as environments, environment configurations and application versions.
Web Server Environment
Uses Route 53, Elastic Load Balancer, Auto Scaling, EC2, Security Groups.
Worker Environment
Uses SQS Queue, IAM Service Role, Auto Scaling, EC2
Elastic Beanstalk Workflow
(1) Create Application (2) Upload application and configuration to Elastic Beanstalk which creates the environment configuration (3) Environment is launched by Elastic Beanstalk (4) The environment can then be managed. If the management of the environment changes the environment configuration, the environment will automatically be updated should additional resources be required.
AWS Lambda charges
You only have to pay for every 100ms of use when the code is running.
Working with AWS Lambda
(1) Upload code to Lambda or write it in the editor provided. (2) Config the code to execute upon trigger from an event source (such as an object being uploaded to an S3 bucket) (3) Lambda will run your code (4) Lambda computes the run time in milliseconds as well as the quantity of lambda functions run to compute cost.
Components of AWS Lambda
Lambda function (compiled of your own code), Event Source (AWS sources that can be used to trigger your lambda functions), Trigger (an operation from an event source that causes the function to be invoked), Downstream Sources (resources required by the lambda function), Log Streams (to identify and troubleshoot issues. They come from the same function and are recorded in Cloudwatch)
Creating Lambda Functions
Select a Blueprint (preconfigured lambda functions to be used as a template), Configure Trigger, Configure Function (upload code or edit in-line, define required resources, max execution timeout, IAM role, handler name)
AWS Lambda benefit
Highly scalable and saves cost
Jobs
Unit of work run by AWS Batch. Can be an executable file, an application or shell script. Run on EC2 instances as containerized app. Has states such as “submitted”, “pending”, “running”, “failed”, etc.
Job Definitions
Specific parameters for the jobs and define how the job will run with what configuration. (ex: how many vCPUs, which data volume, IAM role, mount points)
Job Queues
Jobs that are scheduled are placed into a queue until they run. There can be different queues with different priorities. On-demand and spot instances are supported. AWS Batch can bid on your behalf for spot instances.
Job Scheduling
Takes care of when a job should run and from which compute environment. Typically on FIFO basis. Ensures higher priority queues are run first.
Managed Environments
The service will handle provisioning, scaling and termination of compute instances. Environment is created as an ECS cluster.
Unmanaged Environments
Environments are provisioned, managed and maintained by you. Allows for greater customization but requires greater administration and maintenance. Required you to create necessary ECS cluster.
Amazon Lightsail
VPS (Virtual Private Server). Designed to be quick, simple and easy to use at a low cost point for small businesses and individuals. Usually for simple websites, small applications and blogs. Multiple lightsail instances can run together and communicate. Can connect to other AWS resources and to your existing VPC via a peering connection.
ELB
Evenly distributes requests across EC2 instances, lambda functions, a range of IP addresses or even containers. Targets can be across multiple AZs. ELBs consist of multiple instances so that they are not a single point of failure.
Application Load Balancer
For applications running HTTP or HTTPS. Operates at the request level. Advances routing, TLS termination and visibility features targeted at application architectures allowing you route traffic to difference ports on the same EC2 instance
Network Load Balancer
Ultra high performance while maintaining low latency. Operates at the connection level. Handles millions of requests per second.
Classic Load Balancer
Meant for applications build in the EC2 Classic environment. Operates at both the request and connection level.
ELB Components
Listeners (defines how requests are routed based on ports and protocols set as conditions), Target Groups (resources where request are routed. can route to multiple target groups based on rules), Rules (defines which request is routed to which target group). ELB contains 1 or more listeners. Listeners contain 1 or more rules. Rules contain 1 or more conditions. All conditions in a rule equal a single action. Health Checks (if an instance does not respond to a health check, it stops sending traffic to it). Internet Facing ELB (nodes of the ELB are accessible via the internet so have public DNS name). Internal ELB (can only serve requests from within your VPC.) ELB Nodes (must be defined in every AZ you wish to route traffic). Cross Zone Load Balancing (Ensures that all targets across all AZs have an even distribution)
Using HTTPS as a ALB listener
For an ALB to encrypt traffic it will need an server certification and an associated security policy. SSL is a cryptographic protocol much like TLS. SSL and TLS are used interchangeably when discussing certifications on your ALB.
ALB server certificate
Server certificate used by ALB is a X.509 certificate which is digital ID provided by a Certification Authority such as AWS Certificate Manager (ACM). Used to terminate the encrypted connection received from the remote client and then the request is decrypted and forwarded to the resources in the target group. Can be created and provisioned by either the ACM or the IAM. ACM is preferred. IAM is used in regions not supported by ACM.
Load Balancer OSI Layers
ALB operates at the application layer while the NLB operates at the transport layer. NLB is good choice for high traffic applications or when a static IP is required.
Components of EC2 Auto Scaling
- Create a Launch Configuration or Launch Template, 2. Create an Auto Scaling Group
Block Storage
Data stored in chunks known as blocks. Blocks are stored on a volume and attached to single instance. Very low latency. Comparable to DAS (direct access storage).
File Storage
Data is stored as files with series of directories. Data is stored within file system. Shared access for multiple users. Comparable to NAS (network access storage).
Object Storage
Objects stored across flat address space. Object references by unique key. Each object can have metadata to help catagorize and identify.
S3 Region
Region must be specified when uploading data to S3 but the data will be replicated across AZs.
S3 Bucket
Bucket names must be globally unique. Default limitation of 100 buckets per account but can be increased if requested. Objects have unique object key.. Folders can be useful for categorizing, but S3 operates on the bucket level. It is not a file system.
S3 Storage Classes
Standard, Standard IA (infrequent access), Intelligent Tiering, One Zone IA (infrequent access), Reduced Redundancy Storage (RSS)
S3 - Frequent Access
Standard or Reduced Redundancy Storage (RSS). Standard is default and RSS is old and not recommended.
S3 - Infrequence Access
Standard IA or One Zone IA. Same access speed as Standard. Additional cost to retrieve data. One Zone IA does not replicate data across AZs so should only be used for data that can be reproduced. One Zone IA is more cost effective than Standard IA.
S3 - Intelligent Tiering
Objects are moved back and forth between frequent access and infrequence access depending on access patterns. Great for unpredictable access patterns. Data moved to infrequent access tier if not accessed for 30 days or more. Will be moved to frequent access tier when accessed and the 30 day timer will be reset. There are no retrieval costs like with standard IA and one-zone IA, but there is a cost per object. Each object must be larger than 128kb.
S3 - Bucket policy
Impose set of controls within a specific bucket. JSON. Only controls access to the data in the bucket. Permissions can be very specific (e.g. by user, by time, by IP address), Provides added granularity to bucket access.
S3 - Access control lists
Controls access only for users outside of your AWS account. ACLs are not as granular as bucket policies. Permissions are broad such as “list objects” and “write objects”.
S3 - Data Encryption
Server-side (SSE) and client-side (CSE) encryption methods: SSE-S3 (S3 managed keys), SSE-KMS (KMS managed keys), SSE-C (customer managed keys), CSE-KMS (KMS managed keys), CSE-C (customer managed keys). SSL is used for data in transit.
S3 - Versioning
Allows for multiple version of an object to exist. Useful for recovering from accidental deletions for malicious activity. Only the latest version is shown by default, but it is possible to view all versions. Versioning is not enabled by default. Once enabled, it cannot be disabled - only suspended. Adds a cost because storing multiple versions of objects.
S3 Lifecycle Rules
Ability to move data between storage classes based on specific criteria including Glacier or even deleting the data. The time frame is configurable.
S3 - static content and websites
Any object can be made public and accessible via a URL, CloudFront works closely with S3, Entire static website can be hosted on S3 to make it scalable.
S3 - large data sets
Good for storing large amounts of data. Scalable. Can is accessing simultaneously by different users.
S3 - integrations with other AWS services
EBS uses S3 to backup itself (the backups are not visible to users), Cloudtrail uses S3 to store logs (you can view these S3 obects), CloudFront (S3 can be used as origin for CloudFront)
S3 pricing
Varies by region. RSS is more expensive than Standard. Infrequent Access is more cost effective. The cost per gigabyte reduces when certain thresholds are reached. Additional charges for per 10000 PUT, COPY, POST, LIST requests). Charge for every 10000 GET requests (less expensive). Data transfer into S3 is free but transfer out costs per GB.
S3 anti-patterns
Archiving data for long term use. Data that is dynamic and changes very fast. Data that requires file system. Structured data that needs to be queried.
Glacier vault
container for Glacier archives. Region specific.
Glacier archive
can be any object. a vault can have unlimited archives.
Glacier dashboard
only allows you to create vaults. Operational processes must be done using code: Glacier web service API or AWS SDKs.
Moving data to Glacier
(1) Create vault (can use dashboard), (2) Move data into Glacier using API/SDK or by using Lifecycle rules.
Retrieving data from Glacier
Must use code. Must first create an archival retrieval job. Retrieval options are (1) Expedited: for urgent requests. Must be less than 250MB. Data available in 1-5 minutes. $0.03 per GB, $0.01 per request. (2) Standard: regardless of size. 3-5 hours to retrieve. $0.01 per GB, $0.05 per 1000 requests. (3) Bulk: used to retrieve petabytes of data. 5-12 hours. $0.0025 per GB, $0.025 per 1000 requests.
Glacier Security
Data encrypted by default using AES-256. Also uses vault access policies and vault lock policies.
Vault access policy
Resource based. Applied to specific vault. A vault can only have 1 vault access policy. JSON format. Policy contains principle component (determines “who” has acess). If the user also have identity policy, the vault access and identity policy are both looked at. If either has an explicit deny, access is denied.
Vault lock policy
Once set, cannot be changed. Used to prevent delete of files for compliance reasons.
Glacier Pricing
Single storage cost regardless of how much storage is being used. Varies by region. Transfer in is free. Transfer to another region is $0.02 per GB. There are also charges for retrieval requests.
Benefits of EC2 Instance Store
Included in cost of instance. Very high I/O speeds. Ideal for cache or buffer for rapidly changing data. Often used within a load balancing group where data is replicated for pooled between the fleet.
Instance store volumes
Not available for all instances. Capacity increases with the size of the EC2. Have the same security mechanism as the EC2.
Instance storage anti-pattern
Not to be used for data that needs to remain persisted or needs to be accessed or shared by multiple entities.
Elastic Block Storage
An EBS can only be attached to one EC2 but an EC2 can be attaches to multiple EBSs.
EBS Snapshot
Snapshot can be taken manually or by code. Snapshot is stored in S3. Snapshots are incremental meaning only the data that has changed is covered. New volumes can be created from a snapshot (in case the original EBS is lost). It is possible to copy an snapshot from one region to another.
EBS High Availability
Every write to replicated multiple times with an AZ. If the AZ fails, the EBS data will be lost. You can restore from a snapshot.
EBS SDD
suitable for smaller blocks, databases using transactional workloads, boot volumes for EC2. options are General Purpose SDD (GP2), Provisioned IOPS (IO1)
EBS HDD
designed for workloads requiring high rate of throughput, big data processing and logging information, large blocks of data. Cold HDD (SC1), Thoughput Optimized HDD (ST1)
General Purpose SSD (GP2)
single digit millisecond latency, can burst up to 3000 IOPS, baseline performance of 3 to 10000 IOPS, throughput up to 128 MB/s on volumes up to 170GB, throughput increases to 768 KB/second per GB up to a maximum of 160MB/second (+124 GB volumes)
Provisioned IOPS (IO1)
predictable performance for I/O intensive workloads, specify IOPS rates during creation of new EBS volume, volumes attached to EBS-optimized instances will deliver the IOPS defined within 10%, 99.9% of the time, volumes range from 4 to 16 TB, max IOPS possible is 20,000
Cold HDD (SC1)
lowest cost, designed for large workloads accessed infrequently, high throughput capability, can burst to 80 MB/s per TB, delivers 99% of the expected throughput, can’t be used as a boot volume for instances
Throughput Optimized HDD (ST1)
designed for frequently accessed data, suited for work with large data sets requiring throughput intensive workloads, ability to burst to 250MB/s, maximum burst of 500MB/s per volume, delivers 99% of expected throughput, not possible to use as boot volume
EBS Encryption
EBS offers encryption at rest and in transit. encryption is managed by EBS itself. You just need to specify if you want encryption. Uses AWS-256 by interacting with AWS-KMS (key management service). KMS uses customer master keys (CMK) to create data encryption keys (DEK). Snapshots are also encrypted and well as any volume created from a snapshot. Only available on selected instance types.
Creating new EBS volume
Can be created when creating the EC2 instance or can be created as a stand-alone EBS. For stand-alone, you’ll be asked for the AZ and it can only be attached to instances in that AZ.
Changing size of EBS volume
Can be done in AWS console or AWS CLI. After the increase, you’ll need to extend the filesystem. Also possible to extend by creating new volume from a snapshot.
EBS pricing
charged for storage provisioned (not based on usage), cost varies by volume type and region. Charged on a per second basis. Snapshots are stored on S3 and you will be charged for that storage
EBS anti-patterns
not good for temporary storage or multi-instance storage. Not suited for high durability or availability (S3 or EFS is a better option for this)
EFS
fully managed, highly available and durable, ability to create shared file systems, highly scalable, concurrent access by 1000’s of instances, limitless capacity, regional
Creating an EFS
must select the VPC. AWS will then create mount targets across the AZs. Allows you to connect to mount target IP address. Only compatible with NFS V4.0 and V4.1. Does not support windows OS. Linux instance must have NFS client installed to mount the target. Select performance mode (general purpose or max I/O),, choose general purpose if under 7000 operations is sufficient. (Use the metric PercentIOLimit to see percentage of the 7000 limit used. ) Configure encryption. Data is only encrypted at rest, not in transit. Can connect from on-premise as long as using direct connect or 3rd party VPN.
EFS - general purpose performance
used for most use cases, lowest latency, max of 7000 file system operations per second
EFS - max I/O performance
used for huge scale architectures. concurrent access by 1000’s of instances, can exceed 7000 operations per second, vitually unlimited throughput and IOPS, additional latency per I/O
Moving data into EFS
Can be done security from on-premise or AWS using file-sync agent. On premise can use VMWare ESXi host. AWS can use community AMI to be used with EC2 instance. Migration progress can be monitored with CloudWatch.
EFS pricing
No charge for data transfer. No charge for requests. Charged for data consumption per GB-month.
EFS anti-patterns
Not for data archiving. Not for relational database. Not recommended for temporary storage.
CloudFront
Content Delivery Network (CDN), distributes web traffic closer to end users via edge locations, data is cached (not durable), origin data is S3
Edge Locations
Located in areas of high population. cache data to reduce latency
Web Distribution
distributes static and dynamic content, uses both HTTP and HTTPS, allows you to add, remove and update objects, provides live stream functionality, origin can be web server, EC2 or S3 bucket
RTMP Distribution
For distributing streaming media using Adobe Flash media server’s RTMP protocol. Allows end user to start viewing media before file has been downloaded from the edge location. Source data must be S3.
Distribution Configuration
specify origin location, specify caching behavior options, define edge locations. Options are US/Can/Europe, US/Can/Europe/Asia or All edge locations. Select if should be associated with Web Application Firewall (WAF) for extra security. Can specify encryption via SSL certificate.
CloudFront pricing
Primarily based on data transfer and HTTP requests. Costs also for field-level encryption, invalidation requests, dedicated IP custom SSL
File Gateway
Ability to mount of map drives to S3 bucket as if it was a share held locally. Local cache is used for recently accessed data. Files are stored 1-1 in S3 as objects.
Storage Volume Gateways
Backup you local storage volume to S3. Local data remains on-premise. Mounted as iSCSI devices that applications can communicate with. Data us written to S3 as EBS snapshots. Volumes can be beteen 1GB and 16TB. up to 32 volumes per gateway. Max storage of 512TB per gateway. Storage buffer using on-premise storage is used to stage data. Uploaded using SSL and stored in encrypted format in S3. Easy to create snapshots at any time. Snapshots are incremental to reduce storage costs. If there is a on-premise disaster, EBS volumes could be created from snapshots and applications could be up and running in a VPC.
Cached Volume Gateways
Primary data storage is S3. Local data storage is used for buffering and a local cache for recently accessed data. Presented as iSCSI volumes. Local disks must be selected to be used for buffer/cache. Local disk used as staging point for data to be uploaded to S3. Each volume up to 32TB. Up to 32 volumes. Total storage of 1024TB per cached volume gateway. Possible to create snapshots of volumes as EBS snapshots on S3 which can be used to create EBS volumes in a disaster.
Gateway-Virtual Tape Library
Allows you to backup data to S3 but also use Glacier for data archiving.
VTL Components
Storage gateway: Configured as a tape-gateway acting as a VTL with a capacity of 1500 virtual tapes. Virtual Tapes: equivalent to physical tape cartridge with capacity of 100GB to 2.5TB. Data stored on VTs are backed by S3 and visible in Virtual Tape Library. Virtual Tape Library (VTL): equivalent to tape library containing virtual tapes. Tape Drives: Each VTL comes with 10 tape drives presented as iSCSI devices to your backup application. Media Changer: virtual device presented as iSCSI device to backup applications that manage tapes between your Tape Drive and VTL. Archive: equivalent to off-site storage facility giving you ability to archive tapes from VTL to Glacier.
Gateway pricing
Based on storage, requests and data transfer. Cost affected by region. Transfer in is free.
Snowball
Physical device for transferring on-premise data (petabytes) to S3 or vice versa. Comes as 50TB or 80TB device. Dust, water and tamper resistant. Can withstand a 8.5 G jolt in shipping container. High speed data transfer using RJ45 (Cat6), SFP+ Copper, SFP+ (Optical)
Snowball encryption and tracking
data is automatically encrypted by default using AES-256 using encryption keys from KMS. using end to end tracking using E Ink shipping label which ensures it is sent to the correct facility. Can be tracking using SNS messages or via AWS management console. Is also HIPAA compliant allowing shipping of health data into and out of S3. AWS will remove data from appliance according to NIST standards.
Snowball Data Aggregation
Data can be aggregated across multiple snowballs. As a general guideline, if it will take longer than a week to move data using existing connections, snowball should be considered.
Snowball process
Create an export job, receive delivery of snowball appliance, connect appliance to local network (connect while off, turn on device, configure), ready to transfer data, access required credentials, install snowball client, transfer data using the client, disconnect appliance when transfer is complete, return to AWS using specified shipping carrier. (Snowball appliance is property of AWS)
Snowball pricing
No charge to transfer data in, but you are charged S3 charges. There is a charge for each data transfer job plus shipping costs. 50 TB is $200, 80TB is $250 (Singapore = $320). Allowed 10 days. Delays incur additional charges. Data transfer charges out of S3 vary by region.
Relational vs Non Relational connections
clients to relational db maintain a connection and use SQL. client to non relational use REST over HTTP(S) and client must be authenticated and authorized.
Relational vs Non Relational
Relational: RDBMS/ACID engine, supports complex relationships between tables, uses structured query language, generally accessed using a persistent network connection, uses a schema to define tables. provides a processing engine within the database. Non Relational: simple document or key store, can store many different types, generally accessed using RESTful HTTP, no schema required, every table must have a primary key, scales fast, lighter in design.
DynamoDB
Cloud native database for managing high volumes of records and transactions without the need or provisioning capacity upfront. Fully managed service. Supports both document and key store objects. Runs as a web service. Provides downloadable version that can be run locally or on our server. Supports encryption at rest.
Elasticache
Cache service built from Redis and Memcached database engines. Can sit between web service and database to reduce load on database.
When to use Redis
For features. Need complex types such as strings, hashes, lists, sets, sorted sets and bitmaps. Need persistence of your key store. Need to encrypt your cache data. Need to replicate your cached data. Need automatic failover.
When to use Memcached
For simplicity and speed. Need simplest model possible. Need to run large nodes with multiple cores or threads. Need to scale out/in.
Neptune
AWS native graph database engine. Used for knowledge graphs, recommendation engines, network security, etc. Supported graph models: Property Graph, W3C RDF. Supported languages: Apache, TinkerPop, Gremlin, SPARQL.
Benefits of using AWS RDS
Ability to scale components, automatic backup and patching, highly available (can run in multiple AZs within a region. Oracle, Postgres, MySQL, MariaDB use Amazon’s failover technology. Microsoft SQL uses mirroring. Aurora copies data across clusters in different AZs by default.) Automatic failure detection and recovery.
MySQL
Supports MySQL community edition. Instance familes: micro, general purpose, memory optimized, burst support. Store support: general purpose, provisioned IOPS. Can replicate across AZs. Can increase the size of the database on the fly with no down time. Point in time restore and snapshot features (if plan to use, choose NODP storage engine).
Microsoft SQL Server
Pay as you go. Supports Express, Web, Standard and Enterprise. Instance families: general purpose, memory optimized, burst support. Storage support: general purpose, provisioned IOPS. Can select multi AZ deployments and set automatic backups. Transaction log backed up in 5 minute intervals. Point in time recovery to one given second. Automatic backups can be stored up to 35 days. Can manually snapshot entire database and they will be stored indefinitely.
Oracle
Supports BYOL: SE2, SE1, SE, EE. Oracle Database 12c which includes Oracle Fusion Middleware, Oracle Enterprise Manager. Also supports SAP Business Suite and Oracle, JD Edwards EnterpriseOne, Amazon Redshift as a data source for Oracle Business Intelligence (OBIEE). However you won’t have access to underlying OS.
MariaDB
Community developed fork of the MySQL database. Instance supported: micro, general purpose, memory optimized, burst. Storage: InnoDB is the default storage engine, You can create read replicas.
Postgres
Sits between the cost and scalability of noSQL databases and the power of relational databases. Has achieved lots of certifications and is good for compliance.
Aurora
Amazon’s own fork of MySQL. Built from the ground up to be a fast cloud native database. Replicated data across 3 AZs by default. Instances use a cloud native database cluster that stores the data. Clusters span 2 or more AZs by default. Each cluster will have one read/write instance. Each cluster will have at least one read-replica that supports only read. There can be up to 15 read-replicas per cluster. Makes response and recovery faster than most databases.
AWS Health Dashboards
AWS Service Health Dashboard and Personal Health Dashboard
AWS Service Health Dashboard
provides complete health check of all services in all regions. Service health check statuses: “Service is operating normally”, “informational message”, “service degradation”, “service disruption”. Able to view status for the past year
Personal Health Dashboard
Notifies you of service interruptions that might affect your account. (Service interruptions to systems not used in your account would not be available.) There will be 3 tabs: “open issues”, “scheduled changes”, “other notifications”
What can AWS Config do?
Capture resource changes, act as a resource inventory, store configuration history, provide a snapshot of configurations, notifications about changes, provide AWS CloudTrail integration, security analysis, use rules to check compliance, identify relationships. Does not support all services, but supports the most commonly used. AWS Config is region specific. Services that are global such as IAM can be included in a region’s AWS Config as well.
CloudTrail
Records and tracks all API requests in your AWS Account. Requests can be initiated from SDKs, AWS CLI, AWS Management Console, Another AWS Service. (ex: auto-scaling scaling up or down).
CloudTrail Events
Every API request captured is recorded as an event. Events are recorded in logs and stored in S3. Events contain an array of metadata (Ex: identity of caller, timestamp of request, source IP).
CloudTrail Logs
New log files are created every 5 minutes. Logs are files delivered and stored in S3 defined by you. CloudTrail log files can be delivered to CloudWatch Logs for metric monitoring and alerting via SNS.
CloudTrail Infrastructure
Is a global service supporting all regions. Support for over 60 services and features.
Use cases for captured data
Effective for security analysis. (Monitor restricted API calls. Notification of threshold breaches.) Resolve day to day operational issues: (filtering mechanisms for isolating data. quicker root cause identification. speedy resolution). Able to track changes to AWS infrastructure (Config does this as well and can be integrated). Can be used as evidence for compliance and governance control.
AWS Trusted Advisor
Recommends improvements based on best practices across your AWS account.
AWS Trusted Advisor Catagories
Cost Optimization, Performance, Security, Fault Tolerance
Trusted Advisor and Support Agreements
Not all checks in Trusted Advisor are free. All (50+) are available only if you have a Business or Enterprise support plan. Without the plan, you will only have access to 6 core checks. 6 checks are split between Performance (service limits) and Security (Security Groups - Specific Ports Unrestricted, Amazon EBS Public SnapShot, Amazon RDS Public Snapshots, IAM Use, MFA on root account ).
Trusted Advisor - Business and Enterprise Support Plan Benefits
Able to administer certain functions of Trusted Advisor using the AWS Support API. Able to track the most recent changes to your AWS account by bringing them to the top of your AWS Trusted Advisor dashboard.
Trusted Advisor features available to everyone
Trusted Advisor Notifications (opt-on/opt-out, tracks resource check changes and cost savings estimates over 7 days, email up to 3 recipients), Exclude Items (exclude specific resources in a check, can include it later), Action Checks (links inside a check that lead you to remediate the issue), Access Management (able to manage access down to specific categories, checks and actions using IAM), Refresh (Trusted Advisor automatically refreshes every 24 hours, but you can manually refresh 5 minutes after last refresh. Can refresh individual checks.).
Amazon CloudWatch
Means of monitoring resources via a series of metrics provided by each service
Basic and Detail Monitoring
Default is Basic. Records metrics across your resources every 5 minutes. Detail monitoring records metrics every 1 minute and comes at additional cost. CloudWatch records are maintained for 2 weeks.
Alarm
An alert that is triggered when a threshold is met. Ex: CPU hits 75% so alert triggers auto-scaling or an email.
Alarm States
OK, Alarm, Insufficient Data (usually when alarm was just configured)
CloudWatch Logging
Logs are sent to CloudWatch where they can be viewed or exported.
Reserved IPs in a CIDR block
If 1.0.10.0/24 is selected, the following IPs are reserved: 10.0.1.0 = network, 10.0.1.1 = AWS routing, 10.0.1.2 = AWS DNS, 10.0.1.3 = AWS Future Use, 10.0.1.255 = broadcast
Backup and Restore
Data stored as virtual tape library using AWS Storage Gateway. Use AWS Import/Export to shift large archives. In a disaster, archives are restored from S3 as if we are using a virtual tape. (1) Make sure you have appropriate retention policy for the data. (2) Make sure appropriate security measures are in place for the data. (3) Regularly test the restoration and recovery of the system.
Pilot Light
Data is mirrored and environment is built out as a template (not running). In the event of a disaster, the pilot light environment is scaled up to handle production loads. (1) Setup EC2 instances to replicate or mirror our data. (2) Make sure that we have all custom software packages in AWS. (3) Maintain the AMIs that will be needed for fast recovery. (4) Regularly run and test servers and apply software and configuration changes to match the production environment. (5) Automate provisioning as much as possible using CloudFormation.
Warm Stand By
Run a smaller version of production environment that will be scaled up in the event of an outage. DNS records will be updated to route all traffic to backup environment. (1) Setup EC2 servers to replicate or mirror data. (2) Maintain AMIs as required. (3) Run application using a bare minimum of EC2 instances and infrastructure. (4) Patch and update software inline with our production environment.
Multi-Site
There is a mirror of production in AWS. Setup DNS waiting to route traffic to both sites. In the event of a failover, DNS is updated to route traffic to cutover site. Preferred for fast cutover when cost is not the main constraint.
Failback process
Restoring traffic to the primary site after the primary site after a failover. This means that data replicate will be reversed so that data is not lost while the primary site it down.
Fallback process for backup and restore
(1) freeze data changes to the DR site (2) take a backup (3) restore the backup to the primary site (4) re-point users to the primary site (5) unfreeze the changes
Fallback process for pilot light, warm standby and multi-site
(1) establish reverse mirroring/replication from the DR site to the primary site once the primary site has caught up with the changes (2) freeze data changes to the DR site (3) re-point users to the primary site (4) unfreeze the changes
Synchronous data replication
Data is atomically written to multiple databases. AWS writes data to databases in multiple AZs putting more work on the network.
Asynchronous data replication
Data is written to one database then data is sent from the first database to the replica to be persisted. OK for backup source or read-only reporting databases.
Data Recovery Methods
AWS Import/Export, AWS Storage Gateway,
Well Architected Framework - 5 Pillars
Operation Excellence, Security, Reliability, Performance Efficiency, Cost Optimization
Operational Excellence Pillar
Best Practices: Prepare, Operate, Evolve. Design Principles: Perform operations as code, Annotate documentation, Make small, frequent, reversible changes, Refine operations procedures frequently, Anticipate failures, Learn from all operational failures.
Security Pillar
Best Practices: Identiy and Access Management, Detective Controls, Infrastructure Protection, Data Protection, Incident Response. Design Principles: Implement a strong identity foundation, Enable traceability, apply security at all layers, automate security best practices, protect data in transit and at rest, prepare for security events.
Reliability Pillar
Best Practices: Foundations, Change Management, Failure Management. Design Principles: Test recovery procedure, Automatically recover from failure, Scale horizontally to increase aggregate system availability, Stop guessing capacity, Manage change in automation
Performance Efficiency Pillar
Best Practices: Selection, Review, Monitoring and Tradeoffs, Design Principles: Democratize advances technologies, Go global in minutes, Use serverless architectures, Experiment more often, Mechanical sympathy
Cost Optimization Pillar
Best Practices: Cost-effective resources, Matching supply and demand, Expenditure awareness, Optimizing over time. Design Principles: Adopt a consumer model, Measure overall efficiency, Stop spending money on data center operations, Analyze and attribute expenditure, Use managed services to reduce cost of ownership
Availability Zones
Compute, storage, network and database resources are provisioned within AZs. A single AZ can comprise of several data centers. AZs are linked by highly resilient low latency private connections. Each AZ in a region uses separate power and connectivity providers to isolate AZ failure. AZs will use synchronous replication in multi-AZ database deployments for primary and secondary databases and asynchronous for any read replicas. 3,4 or 5 AZs are joined together private connections.
Regions
Collection of AZs that are close to one another. Region will contain at least 2 AZs. Regions help maintain data compliance laws. Using multiple regions allows an additional level of availability. Some services are region specific, others are not.
Region and AZ naming conventions
Region names can be represented as a friendly name or as a code name. AZs are always code names and starts with the region code name followed by -1a, -1b, etc.
Edge Locations
There are more of these than AZs. Used by CloudFront and AWS Lambda@Edge.
Regional Edge Cache
Sits between CloudFront origin servers and edge locations. Data is retained for longer at the regional edge cache.
Netmask range
between /16 and /28
VPC
Your private CIDR block.
VPC terms
Subnet: subset of the VPC where you can place resources. IGW: Internet Gateway, the Amazon VPC side of an internet connection. Hardware based VPN is the connection between your VPN and the data center. VGW = Virtual private gateway, the VPC side of a VPN connection. CGW -Customer Gateway, the other side of the VPN connection. Routers interconnect subnets and direct traffic between internet gateways, VGWs, NAT gateways and subnets. Peering connections route traffic via private IP addresses between 2 peered VPCs.
VPC endpoint for S3
Enables S3 access without using an internet gateway or NAT. Can us VPC endpoint policies.
Subnets
Subnets are CIDR blocks within the VPC IP range
CIDR blocks
Classless Inter-Domain Routing. A block of IP numbers.
VPC size
You can’t change the size of a VPC after you have created it. You’ll need to create a new VPC and migrate your resources to it.
Subnets and route tables
Each subnet must be associated with a route table. By default this is the main route table. Each subnet must be associated with a Network Access Control List. If one is not specified, it will be associated with the default one.
IGW
Provides connectivity in and out of the VPC. Provides a target in the VPC route tables for internet routable traffice. Provides network address translations for instances that have been assigned public IP addresses. If subnet is associated with route table that has a route to an internet gateway, it is known as a public subnet. That subnet route table needs to contain a route that directs internet bound traffic to the IGW. One way to achieve that is to scope the route to all destinations: 0.0.0.0/0. Or you can scope the route to specific IP addresses.
Direct Connect
You can create a VPN tunnel using IPSEC to on-premise. For a more predictable connectivity or private connection use Direct Connect - a private, dedicated, connection between on-premise and VPC. Uses 802.1 q LAN standard.
Network Access Control Lists (ACLs)
Subnet Group = instance level control. NACL = subnet level control. VPC comes with a default ACL. Allows outbound and inbound traffic by default. Network ACLs are stateless so responses to allowed inbound traffic are subject to the rules for outbound traffic and vice versa. Contains a numbered list of rules that are read in order. Has separate inbound and outbound rules. Each rule can either allow or deny traffic. Can create a custom Network ACL for a subnet. By default each custom ACL denies all inbound and outbound traffic until rules are added (which if different than the default ACL). Each subnet must have a Network ACL. If a subnet is not associated with one, it is associated with the default one. A Network ACL can be associated with multiple subnets, but a subnet can have only one Network ACL. When a subnet is associated with an ACL, the previous association is removed. An ACL can have up to 20 rules. Rules are evaluated in order. Backup layer of defense (after security groups). Good layer for blocking malicious IP / DDOS hosts.
Security Group
Virtual firewall to control inbound and outbound traffic to instances / resources. Can specify allow rules but not deny rules. If an instance is not assigned to a security group, it is assigned to the default security group for the VPC. Security groups are stateful. There are inbound rules and outbound rules. No inbound traffic is allowed until you add inbound rules. Return traffic is allowed by default. Security groups are associated with network interfaces. All rules are evaluated before deciding to allow traffic. First layer of defense. Not good for “blocking” rules.
Stateful vs Stateless
Stateful = return traffic is automatically allowed regardless of any rules. Stateless = return traffic must be explicitly allowed by the rules.
VPC limits
allowed 5 VPCs per region. Default VPC set up in each account. 1 default VPC per region. Has a default IGW, ACL and routing table. Allowed 500 security groups. Allowed 5 IGWs (correlates to the VPC limit per region). Allowed 10 VPNs per VPC. Allowed 200 subnets. Allowed 5 elastic IP addresses.
ELB
Increases availability and fault tolerance. Can balance across AZs. Managed service. Can be internal or external facing. Does not terminate or stop instances. Does not manage scaling. It monitors the health of the instances and directs traffic to other instances if unhealthy. Has ability to support sticky sessions where traffic is routed to the same EC2 using cookies. Traffic is balanced to all instances across all AZs. Supports SS offloading. Supports SSL Termination including offloading of SSL encryption. SSL certification can be managed from inside ELB.
Types of ELB load balancers
Network Load Balancer (For connection base load balancing. For sudden and volatile traffic patterns. Operates at the connection level. Supports static and elastic IPs. Supports routing to multiple ports on the same instance. Good for container services.) Application Load Balancer (Ideal for HTTP requests. Operates at the request request level. Provides more advanced routing capabilities. Multiple ports on a single EC2 instance. Good for when you don’t need static IPs - otherwise use Network Load Balancer.) Classic Load Balancer (permits flexible cipher support. Use case when need a simple load balance or flexible cipher support.)
Choosing an ELB type
If you need to support static or elastic IP address: use network load balancer. If you need control of SSL cipher: use classic load balancer. If you are using ECS: Use application load balancer or network load balancer. If you need to support SSL offloading: use the application or classic load balancer.
“Instance Store” backed instance
Instance built from template stored in S3
“EBS” backed instance
Instance built from EBS snapshot
Stopping an instance store backed instance
You can’t.
Cluster Placement Group
Instances are placed in a single AZ to provide lowest latency between instances. Should use the same instance type and should launch them at the same time.. If an instance fails, it can be restarted but will fail if capacity is not available. If this happens, restart all the instances and they might be migrated to hardware that has capacity.
Spread Placement Group
Spreads instances across underlying hardware. Can spread across multiple AZs. Reduces risk of simultaneous failures. Good for mixing instance types. Can have a maximum of 7 instances per AZ per group. If there is a capacity error, try starting the instance again later.
EIP (elastic IP)
To use a EIP, allocate it to your account then associate it with your instance or network interface. Can allocate up to 5 per VPC. If need more, contact AWS support. When associated with an instance or interface, the instance’s public IP is released. You cannot reuse a public IP address. You can disassociate an EIP from a resource and re-associate with a difference resource. You are not charged for the EIP while the instance is running, but otherwise are (to discourage allocating EIPs and not using them).
SQS
Messages are stored redundantly across servers and AZs. Can handle an unlimited number of messages. With standard SQS, order is not guaranteed and message can be received more than once. With FIFO, precedence can be specified and one time delivery is guaranteed. You can place sequencing information in each message so that they can be reordered after being received. Can set a message visibility window up to 12 hours. Messages an be stored between 1 minute and 2 weeks. Default retention is 4 days. Can trigger autoscaling based on the number of messages in the queue.
Route 53
Supports geographical mapping based on end user’s location. Also has a failover feature that can route users to another endpoint if there is an outage. Does this based on health check endpoints. Enables graceful failover from a dynamic site to a static S3 base site. Can route to the region closest to user. Can route traffic based on percentages.
Route Tables
AWS creates an implicit router during VPC creation process. Default Route Table is known as Main Route Table. Main Route Table cannot be deleted. Subnets will be implicitly associated with the Main Route Table unless a custom route table has been assigned. A subnet can only be associated to a single route table. Multiple subnets can be associated with the same route table. A subnet must have it’s own route to a gateway in order to use it. (e.g. a subnet cannot access the internet because it has a route to a subnet that does have access)
How to change the main route table
Create a new custom route table, associated it with one VPC and test it. In the console set the custom route table as the main route table.
Route table route fields
Destination: the CIDR block for the network you need to route to. Target: the gateway to allow you to get to the destination. Status: e.g. Active, Propagated: Yes/No
Default route
Every route table has a route where the destination is the VPC CIDR block and the target is local. This allows all subnets to communicate with other subnets in the VPC. It is created automatically and cannot be deleted.
Route propogation
When enabled, routes representing your VPN connection over the VGW will be added to the route table automatically.
Autoscaling Launch Configuration
Allows you to specify AMI ID, instance type, key pair, security groups and block device mapping for instances.
Autoscaling Auto Scale Group
Allows you to specify minimum, maximum and desired number of instances. Can specify scaling policy.
Termination Policy
Determines which instance to terminate when scaling down. The default policy is take the AZ with more than 1 instance and terminate the instance from the oldest launch configuration. If the instances were launched from the same launch configuration, then it terminates the instance closest to the next billing hour. Can specify a custom termination policy. Policy is applied to the AZ with the most instances. If the AZs are balanced, then AWS uses the termination policy you specified.
Instance protection
Does not protect an instance from manual termination.
Cloudfront Free Tier
50 GB Data Transfer out, 2 million HTTP/HTTPS requests
Cloudfront Pricing
S3 charges. Transfer from S3 to CloudFront is free. Charges when CloudFront responds to requests. DELETE, OPTION, PATCH, POST and PUT requests to origin are charged. HTTP/HTTPS request are billed per 1000 requests and vary across regions. No charge for 1st 1000 invalidation requests, $0.005 per addition path request. (Invalidation request = request to remove a file from edge location before TTL) SSL (using either SNI) - if dedicated IP customer SSL $600 per month. There is reserve capacity pricing which requires minimum monthly usage level for 12 months. Agreements begin with a minimum of 10TB of data per transfer region.
SNI
Service Name Indication. Relies on the TLS protocol and allows multiple domains to serve content over the same IP address. Older browsers do not support SNI. Alternative is dedicated IP custom SSL.
CloudFront price classes
Price Class All - all edge locations. Price Class 200 - US, EU, Asia, Japan edge locations. Price Class 100 - US, EU edge locations..
CloudFront reports
Cache Statistics, Popular Objects, Top Referrers, Usage Report, Viewers Report
CloudFront Best Practices - Static Assets
Use Amazon S3 to decrease load on web server. Control access to S3 by using origin access identity which will allow only CloudFront to access. Control access to content on CloudFront to private content (e.g. paid subscribers, premium customers) - using signed cookies or signed urls. Edge caching setting high TTLs- do not for headers, query strings or cookies unless absolutely required. Versioning - makes updates and rollbacks easy using filenames to version.
CloudFront Best Practices - Dynamic Content
Cache Everything. Use multiple cache behaviors - only forward only required headers, avoid forwarding all cookies, avoid forwarding agent-user header - instead use is-mobile-viewer, is-table-viewer, etc to differential between device type.
CloudFront Best Practices - Streaming Content
Set the right TTLs - low TTL for the manifest, high TTL for media files and media player. Use HTTP based streaming protocols and distribute via web distributions to deliver multi-bit rate steaming using fragmented streaming format such as smooth streaming.
CloudFront Availability
Design for failure. What is origin fails? Route 53 can perform health checks and then take action. CloudWatch for alarms and notifications. More caching = higher availability.
CloudFront Security
Enable end to end HTTPS. Enable HTTP to HTTPS redirect for each cache behavior. Use AIM users. Use CloudTrail to track changes.
Elastic Beanstalk Pricing
Free to use, but there are charges for the resources used.
Elastic Beanstalk Components
Application version (reference to a section of deployable code, usually a S3 reference) Environment (refers to an application version that has been deployed on AWS resources. Environment includes the AWS resources used by the app), Environment configuration (collection of parameters and settings that determine how resources are provisioned), Environment Tier (refers to how app is provisioned based on what it does (e.g. web service, worker)), Configuration Template (baseline used to create new environment configurations), Platform (OS, programming language, server type and components of Elastic Beanstalk), Applications (collection of environments, environment configurations, application versions)
Elastic Beanstalk - Host Manager
Resides on EC2 instance (web server tier). Aid in the deployment of app, Collate metrics and events from the EC2, Generate instance level events, Monitor both the application log files and the application server, Patch instance components, Manage log files allowing them to be published to S3
Elastic Beanstalk - Daemon
Resides on EC2 instances (worker tier). Pull requests from the queue and sends it to the application. This is why the instance profile role is required so it can read form the queue.
Elastic Beanstalk - customization
Can be done by adding your own configuration files within the application source code. Written as json or yaml and stored in a .config file inside the .ebextensions folder of your source code.
Elastic Beanstalk - Deployment Options
All at once (default), Rolling (deployed in batches), Rolling with additional batch (additional batch of instances added so available is not impacted), Immutable (creates an entirely new instance before terminating old instances)
Elastic Beanstalk - Basic Monitoring
Components send metrics to CloudWatch every 5 minutes. However these are not used to determine health. If an ELB is used, then a health check is sent to all the instances. If there is no ELB, then health is determined by the EC2 status check. (2 types: system status check which detects problems with host and instance status check which detects problems that we will need to investigate)
Elastic Beanstalk - High Level Monitoring
If using ASG, make sure at least 1 instance is running and healthy. Route 53 is configured with correct CNAME to direct traffic to ELB. Will check security group to make sure port 80 is open. For worker environment, make sure that SQS is polled at least once every 3 minutes.
Elastic Beanstalk - Enhanced Monitoring
Displays additional information. EC2s have a health agent running allowing Elastic Beanstalk to capture additional information such as system metrics and web server logs. Data is correlated with data retrieved from ELBs and ASGs. Can also be sent to CloudWatch as custom metrics.
Difference between CloudWatch and Health Agent.
Health Agent can probe at a deeper level and more often. Sends data to Elastic Beanstalk every 10 seconds.
Issues with Traditional Backup Methods
Tapes stored at same location as production, tape can fail, tapes get lost in transit, manual errors. Also scalability, costs and data availability.
Benefits of cloud storage
offsite backup, no scalability constraints, no CAPEX costs, high durability and availability, enhanced security, reliable, zero maintenance of hardware, replication and automation easily configured, readily accessible, easy to test DR plans
S3: Cross Region Replication
Not implemented by default. Copies S3 data to a second region.
S3: Multipart Upload
Recommended for uploads of over 100MB to increase performance of backup process. Breaks large upload into smaller uploads which can be sent in any order. If one of the small uploads fails, only that upload needs to be retried. S3 will reassemble once all have been uploaded. Can pause and resume uploads.
S3: Security
IAM Policies (used to allow and restrict access to S3 buckets and objects at a granular level), Bucket Policies (JSON policies assigned to a bucket. Define who or what can access the bucket contents), Access Control Lists (to control which user or account can access a bucket or object using a range of permissions such as read, write and full control), Lifecycle Policies (to automatically manager and move data between classes based on compliance and governance controls), MFA Delete (ensures a user has to enter a 6 digit code to delete an object to prevent accidental deletes or misuse), Versioning (allows you to recover from misuse or accidental deletion by reverting to older version of object)
DynamoDB API Methods
Managing Tables (ListTables, DescribeTable, CreateTable, UpdateTable, DeleteTable), Reading Data (GetItem, BatchGetItem, Query Scan), ModifyingData (PutItem, UpdateItem, DeleteItem, BatchWriteItem). API is a set of endpoints. This is why it is called a web service. Requests have a signature used for authentication.
DynamoDB keys
a single field used as a key is known as the partition key. when using 2 fields as a composite key, the first key is known as the partition key (or hash key), the second is the sort key (or range key)
Provisioned Throughput
You don’t need to do this for disk space. But you need to reserve read and write capacity. The higher the capacity is set, the higher AWS charges. AWS allows limited burst capacity for occasional bursts of activity. Once burst capacity runs out, a “provisioned throughput exceeded” exception will be thrown.
Read Capacity Unit (RCU)
Will let you retrieve 1 item, up to 4 KB in size, with strong consistency each second. If data is larger than 4KB, then you will use 1 RCU for every 4 KB. These round up so a 5KB item would count as 2 RCUs. If you agree to use eventual consistency, will cost half as much. Recommendation is to use strong consistency for user facing data and eventual consistency for background tasks that scan all the table and can afford to not have the latest.
Write Capacity Unit (WCU)
1 item, update 1KB every second. If item exceeds more than 1 KB, then more than 1 WCU is required.
Setting RCU and WCU
Can be set when creating the table and can be adjusted on the fly for an existing table. Default is 5 RCU and 5 WCU upon creation.
DynamoDB query
Queries returns rows that match a single partition key. If there is no sort key, then there will be at most 1 row. Query can specify a range of sort keys. Can use filters based on any column. Filters are applied after results have been narrowed to 1 partition key. When a filter has to look through 10 rows to determine which 1 to return, it requires 10 RCUs. Can also order results by sort key. Can also limit results returned. Can make strongly consistent or eventually consistent. Strongly consistent will check all 3 replicas and send back to most recent. Eventual will read from 1 replica which might not be the most recent and are faster and cheaper.
DynamoDB scan
Searches entire table across all partition keys. Can be filtered by any attribute. Can be slow and expensive because it requires enough RCU to read the entire table. Cannot be ordered. Always eventually consistent (which lowers cost). Can be run in parallel with multiple threads or multiple servers. Elastic Map Reduce does this.
DynamoDB index
Each query can only use 1 index. The index must be specified in the query. Global Secondary Indexes are used for searches across the entire database. Local Secondary Indexes are used for searches within a single partition key. Indexes use tables containing the index values and the partition key. In the index table, the index is the partition key. Reading from the index table requires RCUs as well. The column values can be placed in the index as well - which is called projecting attributes into the index. By default, all attributes are projected. Can configure what is projected to reduce costs. Sort keys can be added to indexes. DynamoDB will keep indexes updated for you. Because every write to the table means a write to the index, you should have as much write capacity for the indexes as the main table, but only the amount of read capacity as expected.
DynamoDB index
Each query can only use 1 index. The index must be specified in the query. Global Secondary Indexes are used for searches across the entire database. Local Secondary Indexes are used for searches within a single partition key.
Global Secondary Index (GSI)
Created with the table or later. Throughput provisioned separately for each GSI. Limit of 5 GSIs per table.
Local Secondary Index (LSI)
Must be created when the table is created. Throughput is shared with the main table. Limit of 5 LSIs per table.
DynamoDB partitioning
Transparently breaks up large tables onto different servers. Occurs when size exceeds 10GB or read/write capacity exceeds certain limits. The number of partitions is not made visible to users. The number of partitions can be calculated as the larger of table size / 10GB or RCU/3000 + WCU/1000. Care should be taken when doing large imports because allocating high RCU or WCU can cause the table to be partitioned into many partitions before any data has been imported. When a table grows naturally, the partitions will often be a power of 2. When a table is split into partitions, new partitions are created and the old one is removed. Half the partition keys are allocated to one and the rest to the other. (So the data is not necessary split in half, just the keys). A hash algorithm is used to determine which partition to assign the keys. WCU and RCU are divided between partitions. If a partition grows much larger than other partitions, that partition might run out of WCUs or RCUs and calls might fail. Burst capacity might make it more difficult to spot this issue. Should choose a good partition key.
Symmetric encryption methods
AES, DES, Triple-DES, Blowfish
Symmetric vs Asymmetric cyrptography
symmetric = same key is used to encrypt and decrypt. this means that the key would have to be shared and if it is intercepted by a 3rd party, that 3rd party can decrypt the data. symmetric is faster than asymmetric.
asymmetric = uses a private and public key. But are needed to decrypt. The private key is not shared.
Asymmetric cryptography
Someone can send you data using the public key. You can decrypt the data using the private key. RSA - Rivert-Shamir-Adleman, Diffie-Hellman, Digit Signature Algorithm
Encryption Overview
Sensitive data at rest should be encrypted. When moved, should be done using a secure mechanism providing encryption in transit. If encryption in transit is not possible, data should be encrypted prior to transmission. You might need to adhere to specific compliance and legal controls.
SSE with S3
SSE is used to encrypt data at rest. Encrypts at the object level. Encrypted before being written to physical disk. Data is accessed normally as long as you have access to it. Options are: SSE with Amazon-S3 Managed Keys known as SSE-S3, SSE with AWS KMS-Managed keys known as SSE-KMS, SSE with Customer provided keys known as SSE-C
SSE-S3
Amazon S3 uses a unique key to encrypt each data object and this key is encrypted with a master key. Uses AES-256 (a symmetric key). Symmetric encryption works well because AWS manages all access. SSE can work in conjunction with S3 bucket policies. You could enforce “conditions” within a bucket policy to deny objects that are not uploaded with SSE during a PutObject request which would mean that when using the AWS CLI –server-side-encryption ‘AES256’ parameter would have to be used.
SSE-KMS
Can select the default AWS/S3 Customer Master Key (CMK) or one of your existing CMKs. KMS CMK is used to encrypt the data keys, not the actual object itself. Multifaster encryption - the data is encrypted with a data key and then the data key is encrypted with the master key. If you don’t have CMK configured, S3 will create a default AWS/S3 CMK the first time you upload an object. A customer managed CMK give greater flexibility for key management. Supports Bucket Policies with the x-ams-server-side-encryption parameter: “s3:x-amz-server-side-encryption”:”aws:kms”
When you upload an object to S3 using SSE-KMS
A request is made by S3 to KMS -> KMS returns two versions of a data key -> One version is plaintext and used by S3 to encrypt the object -> the other is encrypted and uploaded with the object
To decrypt S3 data using SSE-KMS
S3 sends the encrypted data key to KMS -> KMS uses the CMK associated to decrypt the data key -> KMS responds with the plaintext key -> the key is stored in memory and deleted after decryption has happened
SSE-C
Uses customer provided key. You must provide the key and S3 service itself performs the encryption. Must provide the key with the data object upload using HTTPS. (HTTP will be rejected). After encryption, the AES-256 key is deleted from memory and stores a HMAC value. When accessing the data, you must supply the same key to decrypt the object.
S3 CSE with KMS
You only need to supply the CMK-ID to the S3 encryption client. When you upload the object, a request is made by the client and KMS returns a plaintext and a cipher version of a data key. When you retrieve the object, the client sends the ciphered key to KMS to retrieve the matching plaintext version.
CSE using a custom client-side master key (CSE-C)
Your key is never sent to AWS, so don’t lose it! When uploading an object, you must provide a master key to the client. The master key is used to encrypt a data key generated by the client, which will be used to encrypt the object data. When you retrieve the object, the master key is used to decrypt the data key which then decrypts the object.
Encryption with Amazon Athena
Athena is a serverless interactive query service which uses stand SLQ and executes queries in parallel. It can query S3 data that is already encrypted. It can encrypt the results of the query. The encryption of the results is independent of the underlying queried S3 data. Supports data encrypted with SSE-S3, SSE-KMS, CSE-KMS. Does not support data encrypted with SSE-C, CSE-C. Will only return results for encrypted data in the same region as the query. if you need to query S3 data that has been encrypted using KMS, add the users to the key policy of the CMK to provide access. If you need to restrict specific actions required for Athena, grant access to the following actions: kms:Decrypt, kms:GenerateDataKey.
Elastic Map Reduce (EMR)
Managed service comprised on a cluster of EC2 instances to process and run big data frameworks.
EMR encryption
Can be configured to encrypt data in transit or at rest. Is separate from the EC2 instances so that can be reused for existing and future clusters. By default, data is not encrypted at rest. EC2 instances are created from AMIs. If the root device volume is to be encrypted, must use EMR 5.7.0 or later.
EMR encryption with EBS
If we use EBS as persistent storage, there a number options that can work together: Linux Unified Key Setup (LUKS) - you can use AMS KMS to be used as your key management provider or user Custom Key provider. Open-Source HDFS Encryption - Secure Hadoop RPC - uses SASL, Data Encryption of HDFS block transfer using AES-256
EMR encryption with S3
Encryption at rest: EMR supports SSE-S3 and SSE-KMR for service side encrption.Can use CSE-KMS or CSE-C to encrypt the data before storage. Encryption in transit using TLS certification provider: PEM - you need to create PEM certificates and reference zip file in S3. Custom - you add a custom certification provider as a java class.
EMR / Hadoop encryption
(Once TLD certification provider has been configured) Hadoop MapReduce Encrypted Shuffle uses TLS. Secure Hadoop RPC uses SASL. Data encryption of HDFS Block Transfer uses AES-256.
EMR / Presto encryption
(Once TLD certification provider has been configured) When user EMR version 5.6.0 or later any communication between Presto nodes uses SSL/TLS.
EMR / TEZ encryption
(Once TLD certification provider has been configured) Tez Shuffle handler uses TLS
EMR / Spark encryption
(Once TLD certification provider has been configured) Akka protocol uses TLS. Block transfer service uses SASL and 3DES. External shuffle service uses SASL.
EMR encryption with KMS
Ensure that the role assigned to your EC2 instances within the cluster has the relevant permissions to enable access to CMK -> add the relevant role to the Key users for the CMK.
EMS Transparent encryption with HDFS
Data is encrypted and decrypted transparently without requiring changes to the application code. Each HDFS encryption zone has its own KMS Key, by default EMR uses the Hadoop KMS but you can also select an alternative. Each file encrypted by a different data key which are encrypted which are encrypted with the HDFS encryption zone key; it’s not possible to more files between encryption zones.
RDS encryption
Can be configured to encrypt data at rest during creation. Cannot be configured afterwards. Keys can be issued using KMS using AES-256.
How to encrypt an existing database
Create a snapshot of your unencrypted database. Create an encrypted copy of the snapshot. Use the encrypted copy to create a new database. Your database is encrypted.
RDS - if KMS key is disabled
If the KMS key is disabled, you will not be able to read or write your data and RDS will move its instances into a terminal state. You will need to reinstate the KMS key and recover your database from a backup. Read replicas follow the same encryption pattern as defined by the database source.
Additional RDS encryption mechanisms
Oracle and SQL Server Transparent Data Encryption (TDE), MySQL cryptographic functions, Microsoft Transact-SQL cryptographic functions
Using TDE
To use TDE, the database must be associated with an options group. Option groups provide default settings for your database and help with management. Option groups exist for Oracle, SQL Server, MariaDB, MySQL. You must add the option “Oracle Transparent Data Encryption” to the group. Once added, it cannot be removed.
TDE encryption modes
TDE tablespace encryption which encrypts the entire tables and TDS column encryption which only encrypts individual elements of the database.
Database encryption: RDS
RDS offers encryption across all regions other than China. and only for certain instance types. Applying encryption at rest is simplified thanks to built in application level encryption option (which EMR does not support).
Database encryption in transit
can be done using SSL/TLS. Recommended if you need to abide by compliance or data is highly sensitive. Method in which it is carried out depends on the database engine.
Database encryption with Oracle
Can use Oracle Native Network Encryption (NNE). Will encrypt all connections with the database. It is not possible to use SSL and NNE together. To enable, you must enable NATIVE_NETWORK_ENCRYPTION to the database options group.
Kinesis Overview
Amazon Kinesis Firehose: delivers realtime streaming data to different services within AWS. Fully managed by AWS. Receives data from your data producer and delivers to your destination. Amazon Kinesis Streams: Collects and processes huge amounts of data in realtime. Data can come from a variety of different sources.
Kinesis Firehose Encryption
Data can be sent over HTTPS to Kinesis, but is unencrypted by default. If destination is S3, Firehose can implement encryption using SSE-KMS on S3. Relevant permissions must be assigned to a role for access. (kms:Decrypt, kms:GenerateDataKey). You an apply a policy as a trusted entity on the role itself. It will give Kinesis Firehose the relevant access.
Kinesis Firehose to Redshift or Elasticsearch
Firehose copies data first to S3 as an intermediary action. KMS permissions should be implemented to enforce encryption.
Kinesis Streams Encryption
Amazon Streams has the ability to implement SSE encryption directly from the producers. Both producers and consumers need permissions to use KMS key, otherwise encryption and decryption will not be possible. Producer need kms:GenerateDataKey against CMK and kinesis:PutRecord, kinesis:PutRecords against the Kinesis Stream. Consumer needs kms:Decrypt against the CMK and kinesis:GetRecord and kinesis:DescribeStream permissions against the Kinesis Stream. SSE-KMS for Kinesis Streams gives full at-rest encryption. A new data key is generated every 5 minutes. A small latency of less than 100 microseconds is added to the performance.
Amazon Redshift Encryption
Redshift offers encryption at rest using a 4 tiered hierarchy using either KMS or CloudHSM. When encryption is enabled, it can’t be disabled and vice versa. Once the cluster is encrypted, metadata and any snapshots are also encrypted.
Amazon Redshift Encryption Tiers
Tier 1 = The Master Key. Tier 2 = Cluster Encryption Key (CEK), Tier 3 = Database Encryption Key (DEK), Tier 4 = Data Encryption Keys. The master key can be managed by KSM or CloudHSM. Integration with a HSM device requires additional steps to implement such as adding certifications to establish a trusted connection.
Redshift encryption process (KMS)
(1) Redshift will send a request to KMS for a new KMS Data key. (2) This KMS data key is encrypted with the CMK Master Key. (3) This encrypted KMS data key is then used as the Cluster Encryption Key (CEK). (4) CEK is set to Redshift and stored separately from the Cluster (5) Redshift sends the CEK to the cluster over a secure channel (6) Redshift requests KMS to decrypt the CEK. (7) The decrypted CEF is also stored in memory. (8) Redshift creates a random DEK and loads into memory. (9) The decrypted CEK encrypts the DEK. (10) The encrypted DEK is stored separately from the Cluster (11) The encrypted and decrypted CEK and DEK are stored in memory . (12) The decrypted DEK encrypts the Data Keys generated by Redshift.
CloudHSM Encryption for Redshift
To work with CloudHSM, you must setup a trusted connection between both your HSM client and Redshift using client and server certificates. Using a key pair, redshift created a public client certificate. The certificate is downloaded and registered to the HSM client and assigned to the correct HSM partition. You must configure Redshift with the following details: HSM IP Address, HSM partition name, HSM partition password, Public HSM client certificate.
Redshift Key Rotation
Redshift enables you to rotate keys for encrypted clusters. Cluster will be unavailable during key rotation process. During the rotation, Redshift will (1) rotate the CEK, (2) rotate the DEK, (3) put the cluster into a state of ROTATING_KEYS until the rotation is complete.
IAM User
User objects are created to represent an identity: A user can represent a real person who requires access to operate and maintain your AWS environment. Or it can be an account used by an application that requires permissions to access your AWS resources programmatically.
IAM Groups
IAM Groups are objects like user objects. Groups are not used in authentication process. They are used to authorize access through AWS Policies. Groups contain Users, and have IAM policies associated.
IAM Roles
IAM Roles allow you to adopt a set of temporary IAM permissions. Roles don’t have any access keys or credentials associated with them. The credentials are dynamically assigned by AWS. you can alter the permissions assigned to the Role and all the EC2 instance associated will have the correct access.
IAM Policies
IAM Policies are used to assign permissions. They are formatted as a JSON document and have at least one statement with this structure.
IAM Policy Types
There are two types of IAM policies available: These policies can be associated with Groups, Roles or Users. AWS Managed Policies: Preconfigured by AWS, Most common permissions. Customer Managed Policies: Configured by you
MFA
No additional charge for MFA. you need your own FMA device. It can be a virtual token.
Identity Federation
Identity federation allows you to access and manager AWS resources even if you don’t have a user account with IAM. Identity providers allow users to access AWS resources securely. Other forms of IdP can be any OpenID Connect web providers. Using MS-AD is an effective way of granting access to your AWS resources.
OpenID
Allows authentication between AWS resources and any public OpenID Connect provider such as Facebook, Google, Amazon. When an access request is made, the user IdP credentials will be used to exchange an authentication token for temporary authentication credentials.
SAML
Allows your existing MS-AD users to authenticate to your AWS resources on a SSO approach. SAML lets the exchange of security data, including authentication and authorization tokens to take place between a IdP and a service provider.
Authentication
Your identify + verification (e.g. username and password). Identify must be unique. Another example = credit card + PIN.
Authorization
Authentication takes place before Authorization. Authorization focuses on levels of privileges and permissions to resources.
Access Control
Mechanism (process) of accessing a secured resource: Username/password, MFA. Access controls in AWS: IAM Roles, Federation, NACLs, Security Groups. Access Control is very closely related to both authentication and authorization as the access control mechanism typically uses both to gain access to a resource.
Container Service Characteristics
Run oh separate infrastructure instances, such as EC2. AWS is responsible for managing the OS and the platform. A managed service is provided by AWS fir the actual application which are seen as ‘containers’. You can a number of security reponsibilies (Managing network acess security, Platform level IAM). Example AWS container services include RDS, EMR, Elastic Beanstalk.
Abstract Service Characteristics
Service is removed (abstracted) from the platform or management layer. Access via endpoints using AWS APIs. The underlying infrastructure, OS and platform is managed by AWS. The abstracted service provide a multi-tenant platform of which the underlying infrastructure is shared. Data is isolated via security mechanisms. Strong integration with IAM. Examples of abstract services include: S3, DynamoDB, Glacier, SQS.
AWS Shared Responsibility Model
This dictates which security controls are AWS’s responsibility and which are yours. Boundaries of responsibility will vary between services. It is essential you have clear understanding of where thee boundaries start and end. There are 3 main models that AWS uses to define these responsibilities: Infrastructure Service Model, Container Service Model, Abstract Service Model. As we go from Infrastructure Model -> Container Model -> Abstract Model, AWS assumes more and more responsibility.
Infrastructure Service Model
AWS Responsibility: AWS Foundation Services, AWS Global Infrastructure. Your Responsibility: Customer Data, Platform, Applications, Identity and Access Management. Operating System, Network and Firewall Configuration. Client Side Data Encryption and Data Integrity Authentication, Server Side Encryption, Network Traffic Protection
Container Service Model
AWS responsibility: Platform and Application Management, Operation Systems and Network Configuration, AWS Foundation Services, AWS AIM, AWS Global Infrastructure, AWS Endpoints. Your responsibility: Firewall Configuration, Network Traffic Protection, Client Side Data Encryption and Data Integrity Authentication, Customer IAM, Customer Data.
Abstract Service Model
AWS responsibility: SSE data at rest, Network Traffic protection, Platform & Application Management, Operating System and Network Configuration, AWS Foundation Services, AWS Global Infrastructure, AWS IAM, AWS Endpoints. Your responsibility: Customer Data, CSE an Data Integrity Authentication, Customer IAM.
EMR encryption at rest options
If using EBS, enable local disk encryption within your EMR security configuration. Once that is enabled, you can either encrypt EBS cluster volumes using LUKS or use Open-Source HDFS Encryption using Secure Hadoop RPC or “Data Encryption of HDFS Block Transfer”. If using S3, you can use SSE-S3 … EMR allows you to select encrypt data at rest, in transit or both which can use SSE-KMS or SSE-S3. Can also use CSE with KMS or custom key providers to encrypt your data using your application before it is stored on S3 where it would remain stored in encrypted form (e.g. when using a serializer/deserializer (SerDe) with Hive). Can also do Application Level Encryption and encrypt the entire file using HMAC-SHA1 when storing data on DynamoDB or S3 or Encrypt individual files within your data by using standard SerDe such as JSON for Hadoop.
RDS encryption at rest options
Can enable at creation. Oracle and SQL Server Transparent Data Encryption (TDE).
S3 protecting at rest data options
Implement versioning on the bucket.
Glacier protecting at rest data options
By default, Glacier encrypts data at rest using SSE. If you want another layer of protection, encrypt the data before storing in Glacier.
DynamoDB protecting at rest data options
DynamoDB does not support SSE so encryption of data falls on you. If using java, can use AWS client side to encrypt data. Can also encrypt data with an application development framework before storing your data within DynamoDB.
RDS protecting in transit data
Secure using SSL/TLS. With Oracle, can use NNE instead of SSL.
EMR protecting in transit data
Open-Source HDFS Encryption which provides options of Secure Hadoop RPC or Data encryption of HDFS Block Transfer. Hadoop MapReduce Encryption Shuffle which uses SSL/TLS. There are also options for Spark and TEZ. EMR communications with DynamoDB and S3 are sent over HTTPS. Recommend users connect to EMR cluster for admin purposes using a protocol such as SSH.
Elastic Beanstalk protecting in transit data
Users access web site using HTTP over SSL/TLS (HTTPS) with signed certificates. Traffic between ELB and ELB environment is not encrypted by default but can be achieved.
S3 protecting in transit data
Encrypting in transit is managed by AWS using HTTPS and SSL/TLS connections.
DynamoDB protecting in transit data
When accessing DynamoDB over the internet, the connections should be using HTTPS to ensure the data is encrypted.
AWS Management Control protecting data in transit
The console uses SSL/TLS between your browser and AWS service endpoint in addition to using an X.509 certificate. SDKs, AWS CLI and AWS API calls NOT from AWS Management Console are RESTful APIs over HTTPS.
Layered Network Subnets
(1) Public Subnet: Public internet layer (ELB) (2) Private Subnet: Threat Protection (Security Appliance) (3) private Subnet: Web Server layer (4) Private Subnet: Application Layer (5) Private Subnet: Database Layer. Route tables can be setup to only allow traffic to the layers immediately above and below. NACLs and Security Groups can be implemented more effectively because there will be less ports and protocols to be concerned with in each subnet.
Network Security: EMR
EMR automatically uses some VPC security features. During an EMR job flow, EMR will launch two EC2 security groups: one for the master node and another for the slaves. The master SG allows communication between itself and the EMR service and resources such as S3. The secondary SG allows communication between the slaves and the master. EC2 instances used by EMR within its cluster should be located within a private subnet.
Network Security: DynamoDB, S3 and SQS
Most of DynamoDB security, operational controls and underlying maintenance falls under the responsibility of AWS: DynamoDB hardware failover, Data replication, Network inspections. S3 and SQS network security controls and management for these services are managed by AWS. Controlling who has access to these services is the customers responsibility using IAM.
IAM with RDS
RDS permissions can only be given through IAM Policies (unlike S3 which has bucket policies and S3 ACLs).
IAM with EMR
be default, EMR clusters and restricted to the IAM user who created it. But this can be edited.
S3 Access Control
Can implement additional resource based access control methods. (Bucket Policies, Access Control). Conflicting access permissions between IAM and resource based permissions will be granted on a least privileges basis.
IAM with DynamoDB
With IAM you can grant access to specific rows within a DynamoDB table for specific users giving a fine grained access control on the database itself.
AWS managed CMKs
Used by other AWS services that interact with KMS to encrypt data. They can only be used by the service that created them within a specific region. They are created the first time you implement encryption on that service.
Customer managed CMKs
These provide the ability to implement greater flexibility. You can perform rotation, governing access and key policy configuration. You are able to enable and disable the key when it is no longer required.
Envelope encryption
When one key encrypts another key
Key Policies
These policies allow you to define who can use and access a key in KMS. KMS onfiures the root user of the AWS account full access to the CMK. If the full access user was deleted from IAM, you would need to contact AWS Support to regain the control. Without root access having full access in the key policy, IAM can’t be used for access for others.
Grants
Grants are another method of controlling access and use of the CMKs held within KMS. They allow you to delegate a subset of your own access to a CMK for principles (such as another AWS service within your account). There is less risk for someone the access control permissions for that CMK.
CMK permissions and key policies
Access to CMKs cannot be managed using IAM alone. To manage access to your CMKs, you must use a key policy to your CMK. Without a key policy, users will not be able to use it. Access can be managed using either (1) Key Policies alone (2) Key polices with IAM (3) Key Policies with Grants.
Key Administrators
Creating the CMK through the Management Console gives you the chance to configure different permission sets: * Define the key administrators * Principals can only administer the CMK, not use it to perform any encryption function * You can also specify if you would like them to be able to delete the key * These key administrators have access to update the associated key policy.
CMK Users
Creating the CMK through the Management Console fives you the chance to configure different permission sets: * Define the CMK users * Which users should be allowed to perform any encryption using the CMK. * Users can also use Grants to delegate a subset of the own permissions to another principal, such as a service integrated with KMS or another user.
Using Key Policies with Grants
They allow you to delegate your permissions to another AWS principal within your AWS account. They need to be created using the AWS KMS APIs.
Using Key Policies with Grants
They allow you to delegate your permissions to another AWS principal within your AWS account. They need to be created using the AWS KMS APIs.
Automatic Rotation of CMKs
KMS will rotate your keys every 365 days. The only change is the backing key of the CMK. Older backing keys are retained to decrypt data encrypted before the rotation. If a breach of the CMK occurs, rotating the key will not remove the threat. Automatic key rotation is not possible with imported keys. There is no way to change the time frame of 365 days. However, you an do a manual key rotation. If you key is in a state of disabled for pending deletion, then KMS will not perform a key rotation until the key is re-enabled or the deletion is cancelled. It is not possible manage the rotation of AWS managed CMKs which are rotated every 1095 days (3 years).
Manual Rotation of CMKs
The process of manual key rotation required that a new CMK be created. You will need to update any applications to reference the new CMK-ID. You can use alias names for your keys and update your alias target to the new CMK-ID. You should keep any CMKs that were used to encrypt data before the rotation.
Imported Key Material
Key material is essentially the backing key. When customer managed CMKs are created within KMS, the key material is automatically created for the CMK. To create a CMK without any key material, select “external” for key material origin. The key material can be imported from your own on-premise key infrastructure.
How to import key material
Download a wrapping key (public key) and an import token. (1) AWS KMS provides a means of encrypting it with a public/wrapping key with either a) RSAES_OAEP_SHA_256 (preferred) b) RSAES_OAEP_SHA_1 c) RSAES_PKCS1_V1_5. The import token is used when uploading your encrypted key material. Both the public wrapping key and import token is only active for 24 hours. Encrypt your key material. The key material must be a binary format to allow you to use the wrapping key. Select your CMK. Select to import the key material along with the location of the import token. (Optional) set an expiration of the key material being imported.
Key Material Considerations
The key material created by KMS for CMK has a higher durability and availability. You can set an expiration for imported key material (expired key material is deleted by KMS). In a region wide failure, you will need to have the key material to import back into the CMK.
Deleting a CMK
KMS enforces a scheduled deletion process which can range from 7-30 days. The CMK is taken out of action and put in a state of pending deletion. Keys in this state cannot be used for encryption or decryption and cannot be rotated. You can analyze CloudTrail logs to see when events on the CMK occurred. AWS recommends that you set up a CloudWatch alarm if anyone tried to use the key to encrypt or decrypt with the CMK. If you are not confident that the CMK is no longer in use, you can disable the key. If the CMK uses import material, then just the imported material an be deleted as well as the CMK itself.
EC2 - Payment Options
On Demand = pay hourly. Spot Pricing = bid. Reserved Instances = discounted hourly pricing for upfront commitment. Scheduled Instances = reserve capacity in advanced for jobs that reoccur but do not run continuously. Linux and Ubuntu support per-second billing for on-demand, spot, reserve instances with minimum billing time of 60 seconds. Other instance types are billed per hour.
Instance Type choices based on cost
For applications that benefit from low cost per CPU, use compute optimized instances first. For applications that would benefit from low cost for GB of memory, use memory optimized. If running a database, use EBS optimization or instances that support placement groups. For applications with high internode network requirements, choose an instance that supports enhanced networking. Use placement groups for applications that require low network latency for high network throughput.
Autoscaling spot instances
Bid price is set in launch configuration. Cannot use the same launch configuration for spot and on-demand instances.
Hourly billing
Partially used hours are changed for the entire hour. billing stops when you terminate or stop an instance. Every time you stop/terminate and then start an instance, you will be charged for at least an hour each time even if done multiple times in an hour.
On reboot
host computer stays the same. private and public IP addresses stay the same. EIP remains associated with the instance. Data on instance store volumes are preserved. Root device volume is preserved. The instance billing hour does not change.
on stop/start (EBS backed instances only)
instance runs on a new host computer. For EC2-classic, host gets a new private and public IP address. For EC2-VPC, host keeps its private IP address but gets a new public IP address unless using EIP. For EC2-Classic, EIP is disassociated from the instance. For EC2-VPC, EIP remains associated with instance. Data is erased on instance store volumes. Root device volume is preserved. You stop incurring charges when stopped, but a new billing hour is started upon start.
on terminate
EIP is disassociated from the instance. Instance store volume is erased. Root device volume is deleted by default. You stop incurring charges as soon as state changes to shutting-down.
consolidated billing
Allows for billing of multiple accounts to be combined. Setup using email between account owners. Not related to VPC peering, IAM access, etc.
Transfer Acceleration
Can speed up transfer rate of data from distributed locations over the public internet. Uses multi-part upload and comes with a small additional charge. Can be used on a direct connection, but it designed for use on public internet.
Optimizing mixed requests from S3
Introduce some randomness in the key name to prevent items from overloading a single partition. Example - add a random string prefix to the name. Only applies if workload exceeds more than 100 requests per second.
