AWS Certified Developer Associate Flashcards
Kinesis: What is it? can it be used for real-time operations? what two operations are used to write records into Kinesis and how do they work?
Kinesis - stream data operations
can be used for real-time applications
writing records to Kinesis:
PutRecord: writes a single record to the stream
PutRecords: writes multiple records to the stream in a batch. a single failure in one being written does not halt the entire operation.
Kinesis Data Firehose: what does it do? where does it store data?
Kinesis Data Firehose
captures streaming data
can excrypt, transform, batch, convert it to a columnar data format, or perform other operations on it before storing it
stores data into S3, RedShift, ElasticSearch, or Splunk
*sink types
Kinesis Data Analytics
Kinesis Data Analytics
allows you to run SQL queries on stream data
Kinesis data streams: what does it do? what is the kinesis agent? what is the kinesis producer library? how do you resolve a provisioned throughput capacity error in kinesis data streams? what is a partition key? what does it mean for the partition key if you are getting ProvitionedThroughputExceeded errors?
Kinesis Data Streams
collects huge amounts of streaming data in real time from website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.
enables real-time dashboards, real-time anomaly detection, dynamic pricing, etc.
Kinesis Agent:
stand-alone Java software application that offers an easy way to collect and send data to Kinesis Data Streams.
Kinesis Producer Library (KPL):
The KPL is an easy-to-use, highly configurable library that helps you write to a Kinesis data stream
resolve a ProvitionedThroughputExceeded exception:
to resolve a ProvitionedThroughputExceeded exception, configure the producer to retry with exponential backoff and increase the number of shards within your data strams to provide enough capacity
partition key: used by Kinesis data streams to distribute data across shards. if you have ProvitionedThroughputExceeded errors, your partition key probably isn’t distributed enough
if you are well below you provisioned throughput capacity but still receiving
kinesis video steams
Kinesis Video Streams
enables you to stream video data from connected devices to aws
enables video playback (enables live and on-demand video playback), storage (storage, indexing, and encryption of video data), analytics/machine learning (take advantage of AWS Rekognition and other ML libraries).
Kinesis Adapter
Kinesis Adapter
recommended way to consume streams from DynamoDB for real-time processing
OpenSearch Service
OpenSearch Service
Analyze and monitor activity logs, data from aws services (cloudwatch logs, kinesis data streams, dynamodb), product usage data, CRM data, social media sentiments, and mobile app usage
Glue
Glue
Point glue at data you have stored in aws and glue will discover it and store the metadata, at which point it is ready to have queries run on it
AppSync
AppSync
handles all data-driven application management tasks (online and offline data access, data synchronization, and data manipulation across multiple data sources.)
uses graphQL
EventBridge
EventBridge
responds to event sources like ZenDesk or Shopify and forwards them to targets like Lambda or Saas applications
SNS: what is it? what are filter policies? what are sns topics? what are the types of sns topics and the differences between them?
SNS
fully managed pub/sub messaging service
can send real-time messages to services
filter policies:
by default subscribers receive all messages from publishers. filter policies can be placed on topics, which is a json policy that limits which messages the subscriber receives
topics:
logical access points for producer systems to send messages across to message consumer systems
can be a standard or FIFO topic (name/topic type can’t be changed once created)
you subscribe to a topic to receive messages from it
*configuring topics so that lambda functions can communicate with them and they can send messages to different people
SQS: what is it? what do the DeleteQueue, RemoveQueue, PurgeQueue, and RemovePermission api calls do? what are backlog per instance variables? what are the limits to the amount of messages that can be in an sqs queue? what are the differences between dead letter queues, FIFO queues, standard queues, and delay queues? what is the difference between long polling and short polling? what is the message size limit, and how can you send a larger message? what must you do to enable scaling of sqs? how is encryption acheived in sqs?
SQS
fully managed message queueing service
DeleteQueue, RemoveQueue, RemovePermission, PurgeQueue api calls:
DeleteQueue: Deletes the queue specified by the QueueUrl, regardless of the queue’s contents
RemovePermission: Revokes any permissions in the queue policy that matches the specified Label parameter
PurgeQueue: Deletes available messages in a queue (including in-flight messages) specified by the QueueURL parameter
CreateQueue: creates a new standard or FIFO queue (defaults to standard). can’t change the queue type after creating it. visibility timeout default is 30 seconds
backlog per instance variables
backlog per instance: used instead of ApproximateNumberOfMessagesVisible as a metric for an EC2 AG autoscaling metric
limit to the amount of messages in an SQS queue:
the message capacity in an sqs queue is unlimited
*delay queues, dead-letter queues (and when will SQS add a message to a dead-letter queue), FIFO queues, standard queues
Delay queues: let you postpone the delivery of new messages to consumers for a number of seconds. can delay from 0-15 minutes (DelaySeconds parameter). useful when a consumer needs additional time to process messages
Dead-Letter Queues: where other queues can send their messages after messages are processed unsuccessfully. useful for debugging
FIFO Queues: high (but limited) throughput, messages sent exactly once, and ordering is exact (single-lane highway)
Standard Queues: unlimited throughput, messages sent are delivered at least once, and ordering is best-effort (many-lane highway)
*long polling vs short polling
long polling: SQS returns the messages queried, or waits until messages appear in the queue if no messages are present (this is almost always preferable as is reduces calls to query the queue)
short polling: SQS returns the messages queries, or return automatically if no messages are present (this can easily exceed throughput)
*sending a message larger than 256kb
must use the sqs extended client for java
*know that SQS scales automatically and that nothing has to be done for scaling
SQS KMS: allows messages sent to SQS to be encrypted, with a key managed by AWS KMS
max message size: 256KB
step functions: what are they? what are the states? what are the two different workflows available?
Step Functions
coordinate the components of distributed applications and microservices using visual workflows
provides a graphical console to arrange and visualize the components of microservice application
automatically triggers and tracks each step, and retries when there are errors
logs each steps state for debugging
*know each type of state (success, fail, etc.)
pass: passes input to output without doing any work
task: represents a single unit of work performed by a state machine. uses an activity or aws service
choice: represents a branch in the state machine that may pass data to one of many choice states
wait: delays the state machine from continuing until a specified time
success: represents the successful execution of the step function
fail: represents the failed execution of the step function
parallel: can be used to evaluate separate branches of execution in parallel in the state machine
map: used to run a set of workflow steps on each item in a dataset. runs in parallel
*Standard Workflows vs Express Workflows and the uses cases for each
standard (default): ideal for long-running (up to one year), durable, and auditable workflows. can run for up to a year.
express: ideal for high-volume, event-processing workloads such as IoT data ingestion, streaming data processing and transformation, and mobile application backends
EC2: what is it?
EC2
VPS in the cloud
EC2 autoscaling: can AGs span regions?
where are new EC2 nodes launched when an AZ containing EC2 nodes in an autoscaling group becomes unhealthy? An Auto Scaling group has a maximum capacity of 3, a current capacity of 2, and a scaling policy that adds 3 instances. what is the outcome of the scaling policy? cloudwatch metric integration: what type of metrics exist for AGs?
Autoscaling
can autoscaling groups span regions?:
AGs can span AZs, but not regions
where are new EC2 nodes launched when an AZ containing EC2 nodes in an autoscaling group becomes unhealthy?
Autoscaling attempts to use instance distribution to spread instances in an AG group as far across AZs as it can
if an autoscaling group is provisioned to spread across 3 AZs and 2 instances are added, it will provision those instances in 2 of the 3 AZs
An Auto Scaling group has a maximum capacity of 3, a current capacity of 2, and a scaling policy that adds 3 instances. what is the outcome of the scaling policy?
1 instance is added
cloudwatch metric integration
cloudwatch metrics exist for AG groups including GroupMinSize, GroupMaxSize, GroupTotalInstances, GroupPendingCapacity, WarmPoolPendingCapacity, etc.
EC2: what are the instance types that exist?
instance types (on-demand, dedicated hosts, dedicated instances, spot instances)
on demand: Pay, by the second, for the instances that you launch.
savings plans: reduce costs by making a usage agreement for 1 or 3 year periods
reserved instances: make a commitment to a consistent instance configuration, including instance type and Region, for a term of 1 or 3 years.
zonal reserved instances: reserved instances specific to a certain availability zone. eligable for reserved instance discounts and a capacity reservation
regional reserved instances: a reserved instance for a specific region
spot instances: Request unused EC2 instances, which can reduce your Amazon EC2 costs significantly
dedicated hosts: Pay for a physical host that is fully dedicated to running your instances, and bring your existing per-socket, per-core, or per-VM software licenses to reduce costs.
dedicated instances: Pay, by the hour, for instances that run on single-tenant hardware
EC2: user data: what is it? what types are there?
details on EC2 user data
perform common automated configuration tasks and even run scripts after the instance starts.
add users, groups, install/update packages, start/stop systemd services, create simple web pages, modify file ownership/permissions
types…
shell scripts:
cloud-init directives:
EC2: security groups: what are they? are they stateful or stateless?
control inbound/outbound traffic to an ec2 instance
stateful: if a requests is allowed into a security group, then the response generated from that request is allowed out of the security group regardless of any potential outbound rules on the group.
EC2: do instance key pairs need to be created by the root user? how are you charged for reserved instances based on how much you use them? how do you import the same ssh key into multiple regions? EC2 T family: how is someone charged for using 35 seconds of a burstable instance over the course of a month?
know that EC2 instance key pairs do not need to be created by a root user
how is someone charged for using 35 seconds of a burstable instance over the course of a month?
burstable instances:
T instance family. reduced CPU
provides a baseline CPU performance with the ability to burst above the baseline at any time for as long as required
you can use T2.micro burstable instances for free within certain usage parameters if your account is less than 12 months old
how are you charged for reserved instances based on how much you use them?
you are charged for reserved instances independently of usage
are can apply reserved instance billing to an instance, but running instances concurrently will only apply said billing benefit to a single of the many concurrently running instances, the others will run with on-demand pricing
reserved instance benefits are applied to a maximum of 3600 seconds per clock hour
importing the same ssh key into multiple regions
generate a public ssh key (.pub) from a private ssh key (.pem), then, select the aws region you want to import it into and import it
*max IOPS for general purpose EC2 instance ssd volume
*vallid GiB size and 15000 IOPS configuration for Provisioned IOPS SSD (io1) volume
EC2: what is elastic IP?
Elastic IP Address:
static, region specific IP address
allocated to a specific resource (EC2 instance, network interface, etc.)
Elastic Beanstalk: what is it?
Elastic Beanstalk
tool for deploying and scaling web application
for apps written in Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on Apache, Nginx, Passenger, etc. servers
upload application code, and Elastic Beanstalk handles deployment, capacity provisioning, load balancing, auto scaling and application health monitoring
Elastic Beanstalk: what are the deployment methods?
*what deployment methods are available?
All at once – Deploy the new version to all instances simultaneously. All instances in your environment are out of service for a short time while the deployment occurs.
Rolling – Deploy the new version in batches. Each batch is taken out of service during the deployment phase, reducing your environment’s capacity by the number of instances in a batch.
NOTE: in rolling deployment, instances are not actually lost, just divided into groups and deployed in batches
Rolling with additional batch – Deploy the new version in batches, but first launch a new batch of instances to ensure full capacity during the deployment process.
Immutable – Deploy the new version to a fresh group of instances by performing an immutable update. EBS creates an autoscaling group behind your load balancer and creates a single instance with the new application in the group.
once the new instance passes the health checks, more instances are added to the new AG until the amount of applications in the new AG equal that of the original AG. once these instances pass health checks,
the new instances are transfered to the old AG. the old instances and temporary AG are terminated
Traffic splitting – Deploy the new version to a fresh group of instances and temporarily split incoming client traffic between the existing application version and the new one.
blue/green - create a new environment (a duplicate of the old one) and change the CNAMES of the environments, whiching traffic at the load balancer to redirect traffic instantly
linear deployment - traffic is shifted in equal increments with an equal number of minutes between each increment
canary deployment - the new version of an application is deployed and traffic is randomly directed between the old and new version, according to a preconfigured ratio. this continues until confidence is gained in the new application and traffic is shifted completely over
Elastic Beanstalk: what is .ebextensions? what is the naming convention of files under .ebextensions? what will happen to resources created as part of your .ebextensions if the environment is terminated?
what is .ebextensions/? what happens to applications there when the environment is deleted? what is the config file naming convension in ebextensions? (.ebextension/.config)
directory for EBS econfiguration
files in this directory follow the naming convention .ebextension/*.config
Any resources created as part of your .ebextensions is part of your Elastic Beanstalk template and will get deleted if the environment is terminated.
Elastic Beanstalk: what will happen to instances that failed to deploy correctly after being manually terminated?
*status of instances of an application that failed to deploy correctly after being manually terminated
elastic beanstalk will replace the instances with instances running the application version of the most recent successful deployment
Elastic Beanstalk: what are the different environments?
*environments (dedicated worker instance, load-balanced, autoscaled, etc.)
web server environment:
load-balanced, scalable: A load-balanced and scalable environment uses the Elastic Load Balancing and Amazon EC2 Auto Scaling services to provision the Amazon EC2 instances that are required for your deployed application
single instance: contains one Amazon EC2 instance with an Elastic IP address. use a single-instance environment if you expect your production application to have low traffic or if you are doing remote development
worker environment: run long-running tasks that a web back-end would be expected to run
Elastic Beanstalk: how do you configure https?
HTTPS in Elastic Beanstalk
you can use https in your elastic beanstalk environment by assigning a server certificate to your environments load balancer
you can configure a proxy server behind your load balancer that runs on your instances to terminate https if you need https all the way up to your EC2 instances or are running a single instance environment
you can also combine https terminated at the load balancer and https terminated at the instances for https end to end in a load balanced environment
forcing https in a load balanced environment: open ports 443 and 80 on the load balancer, at the ssl cert to the load balancer, have the load balancer forward http and https traffic, have the instances redirect http traffic to https
Elastic Beanstalk: what are the config files involved?
cron.yaml:
a file that can be defined in an EB worker environment to perform a set of repetitive tasks scheduled asynchronously
env.yaml:
environment manifest
to configure the environment name, solution stack and environment links to use when creating your environment
Elastic Beanstalk: what are lifecycle policies?
Lifecycle Policies:
Each time you upload a new version of your application with the Elastic Beanstalk console or the EB CLI, Elastic Beanstalk creates an application version. If you don’t delete versions that you no longer use, you will eventually reach the application version limit and be unable to create new versions of that application
You can avoid hitting the limit by applying an application version lifecycle policy to your applications. A lifecycle policy tells Elastic Beanstalk to delete old application versions or to delete application versions when the total number of versions for an application exceeds a specified number.
Elastic Beanstalk: what are platforms?
platforms
Elastic Beanstalk provides managed platforms that support running web applications developed for specific programming languages, frameworks, and web containers.
This includes platforms for running applications written in specific languages like Java, Go, or Python, or applications using certain technologies like Docker or Tomcat
Lambda: what is it?
cloud function invokation service
Lambda: layers
lambda layers
zip file containing supplimentary code and data (custom runtime, library funtions, etc.)
can only be used with functions deployed as zip archives
changes pushed to layers create a new version of the layer. each new version is considered a new entity with a unique ARN
zip file archive that contains supplementary code or data. Layers usually contain library dependencies, a custom runtime, or configuration files
can have up to 5 layers
each layer can be up to 250MB
can be common to, shared by, and pushed to many different functions
Lambda: function memory/cpu allocation
function memory allocation: know that cpu power is allocated proportionally to memory allocation, know that increasing memory allocation can decrease runtime for workloads with significant CPU utilization
Lambda: concurrency: reserved concurrency, provisioned concurrency, concurrency scaling rate, what error occures if the requests outpace this rate, using unreserved concurrency
reserved concurrency: upper limit to the amount of instances of a function that can be running at a given time. free of charge
provisioned concurrency: certain amount of pre-warmed, provisioned instances of the function that exists and are always ready to respond to invokations. incur additional cost
concurrency scaling rate: Your concurrency scaling rate is 1,000 execution environment instances every 10 seconds per function Every 10 seconds, Lambda can allocate at most 1,000 additional execution environment instances to each of your functions. requests that outpace your concurrency return a 429 error using unreserved concurrency: You can reserve up to the Unreserved account concurrency value minus 100. The remaining 100 units of concurrency are for functions that aren't using reserved concurrency (functions that are using unreserved account concurrency). For example, if your account has a concurrency limit of 1,000, you cannot reserve all 1,000 units of concurrency to a single function. https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html
Lambda: how does function monitoring work?
logging events and getting access to them via cloudwatch logs
Lambda automatically integrates with CloudWatch Logs and pushes all logs from your code to a CloudWatch Logs group associated with a Lambda function
Lambda: environment variables: what are they? what is the format of predefined variables? what is the size/amount limit quota? can you encrypt them? if so, how is that achieved?
environmental variables:
You can use environment variables to adjust your function’s behavior without updating code.
An environment variable is a pair of strings that is stored in a function’s version-specific configuration.
The Lambda runtime makes environment variables available to your code and sets additional environment variables that contain information about the function and invocation request.
predefined environmental variables are in ALL_CAPS_UNDERSCORE_NOTATION
environmental variables are encrypted by default via AWS KMS keys (AWS managed KMS keys, by default)
total size/amount of environmental variables in AWS Lambda (total size can’t exceed 8KB, no limit to amount of variables)
Lambda: what is the /tmp directory? what are the size restrictions of it?
/tmp directory:
This is 512MB of temporary space you can use for your Lambda functions.
configurable from 512MB and 10,240MB, 1MB resolution
Lambda: what are event source mappings? what services are they available for?
event source mapping:
An event source mapping is a Lambda resource that reads from an event source and invokes a Lambda function.
You can use event source mappings to process items from a stream or queue in services that don’t invoke Lambda functions directly.
This page describes the services that Lambda provides event source mappings and how-to fine tune batching behavior.
available for DynamoDB, Kinesis, MQ, apache Kafka, SQS, documentDB
Lambda: what is the timeout upper/lower bounds?
timeout:
max time the lambda function will run
default 3 seconds
ranges from 3 seconds to 15 minutes (1 second resolution)
Lambda: running a lambda function in a VPC
running a lambda function in a VPC
by default, lambda functions are not connected to VPCs.
if you connect them to VPCs, the function cannot access the internet by default.
Lambda offers hyperplane ENIs, which are managed resources that provide connectivity to the internet to lambdas in VPCs.
configuring lambda to access private resources in a VPC: Lambda will set up one hyperplane ENI per resource you are accessing in the VPC. if a function needs the internet, you have to set up an internet gateway for traffic to get from the VPC out to the internet, because configuring an ENI for a lambda function prevents it from having access to the public internet (REGION lambda ---(--VPC--(--AZ--(--PRIVATE SUBNET--> ENI--> instance--)--(--PUBLIC SUBNET--> NAT GATEWAY--)--)--)--> INTERNET-GATEWAY ) The lambda must connect to an ENI in a private subnet, because you cannot get a public IP for the function to connect to you must get out of the VPC via a nat gateway that connnects to an internet getway. You must do this if you want internet connection you need the ec2:CreateNetworkInterface, ec2:DescribeNetworkInterface, ec2:DeleteNetworkInterface permissions, all included in the AWSLambdaVPCAccessExecutionRole managed policy
Lambda: what is RDS proxy?
RDS proxy:
You can connect a Lambda function to an Amazon Relational Database Service (Amazon RDS) database directly and through an Amazon RDS Proxy.
Direct connections are useful in simple scenarios, and proxies are recommended for production.
Lambda: how to configure EFS for lambda
EFS For Lambda:
create a mount target for EFS in every AZ that your lambda function connects to
ensure port 2049 is open on the security groups used by the function and mount targets
an access point connects each instance of the function to the right mount target for the AZ it connects to
for best performance, create an access point with a nonroot path and limit the amount of files you create in each directory
Lambda: what are they? what can they point to? can you split invokation requests issued to aliases to 2 functions? what are the requirements of doing this?
aliases
an alias is a pointer to a function with a unique ARN
aliases can only point to functions, not other aliases
you can update an alias to point to a new version of a function
you can use aliases in the mappings from events to functions that event sources like S3 maintain. when the function version changes, the mapping doesn’t have to be updated
you can configure aliases to send a certain degree of invokations it gets issued to one function it points to and the rest of the invokations it gets issued to a secondary function both functions must be published, the alias cannot point to $LATEST, and must have the same execution role (and deadletter queue configuration, if present)
Lambda: versions: what is it? what is $LATEST? what components make up a function version? qualified vs unqualified ARNs? what types of deployments do they support?
versions:
every time you publish changes to a lambda function, a new version is created.
the latest version is unpublished, and referred to by the name $LATEST.
function versions are immutable: you cannot change them once published
a function version is composed of: the function code, dependencies, runtime identifier, runtime version, function settings, environmental variables, and a unique ARN the version of a function can optionally be placed after the ARN of the function (this is referred to as a qualified ARN): ARN of version 42 of the helloworld function: arn:aws:lambda:aws-region:acct-id:function:helloworld:42 the unqualified ARN implicitely points to the $LATEST version you cannot create an alias with this version supports blue/green and rolling deployments
Lambda: how are permissions configured?
permissions
you can grant permissions to a function with resource based and identity based policies
the scope of the permission depends on if you grand permissions to a function, alias, or version of the function
Lambda: how do you deploy code to lambda?
deploying code to lambda
you upload code to lambda in a deployment package, of either a zip archive or container image type
code must be packaged into a deployment package whether it is uploaded via the lambda console or a toolkit, Lambda API, command line tools, or the AWS SDKs
you cannot change the deployment package type of a function
Lambda: zip archives
zip archives
Lambda functions with function code written in a scripting language, the deployment package must be less than 3MB in size, and has no library dependencies can be edited in the embedded code editor
other code must be uploaded in a zip archive.
if the zip archive is less than 50MB in size, it can be uploaded from your local machine, else, it has to be uploaded via an S3 bucket
unzipped size cannot exceed 250MB, including layers and extensions
Lambda: container images: what are they? max uncompressed size, layer support, function stages and optimization, stage transition after inactivity, creating a container image function, base image types, multi-architecture container image support, runtime api implementation, OS-based container image support
container images
max uncompressed size: 10GB
you can invoke functions in container images during the pending state, however, other function operations may fail during pending, the function is optimized. if the function isn't invoked for multiple weeks, the functoin state moves to inactive, and the optimized version is reclaimed does not support layers creating a function from a container image build the image locally upload to ECR specify the repository URL when you create the function (you can create a function using an image from a different account as long as it is in the same region as the function) base image types: AWS base image: a runtime runtime interface client (to manage interaction between lambda and your function code) and a runtime interface emulator (for local testing) AWS OS-only base image: Amazon Linux disto and a runtime interface emulator (for local testing) commonly used to create images for compiled language (Go, Rust, etc.), or for a language that Lambda doesn't provide a runtime for. not pre-loaded with a runtime interface client (hence the name) you must include a runtime interface client non-AWS base image: base image made from another container registry (alpine linux, debian, etc.)
lambda container images:
Lambda does not support functions that use multi-architecture container images
the container must implement the Lambda runtime api
supports only linux-based container images
Lambda: runtimes: what are they, how does selecting a runtime differ based on the packaging type?
runtimes
how lambda supports multiple languages
each language has its own runtime and unique runtime identifier
for container images, you choose the runtime when you create the image
for zip archives you choose the archive when you make the function
Lambda: test events
test events
a test event is a json input to your function
private: Private test events are available only to the event creator, and they require no additional permissions to use. You can create and save up to 10 private test events per function.
sharable: Shareable test events are test events that you can share with other users in the same AWS account.
Lambda: invokation styles
sychronous invokation
when you invoke a lambda function synchronously, lambda starts the function and then waits for the result, blocking as it waits
asyncronous invokation (X-Amz-Invocation-Type: Event)
many services invoke lambda functions asynchronously.
in this case, lambda queues the invokation request(s) before sending them to the function
Lambda retries function errors twice. If the function doesn't have enough capacity to handle all incoming requests, events might wait in the queue for hours or days to be sent to the function. You can configure a dead-letter queue on the function to capture events that weren't successfully processed. Async invokations support failed requests being sent to a dead-letter queue (SNS topic or SQS queue) that receives events for troubleshooting or reprocessing
Lambda: event filtering, what is it, what services are they available for, how many filters can exists on one source mapping, and how are multiple filters on one source mapping combined?
event filtering
You can use event filtering to control which records from a stream or queue Lambda sends to your function
available for available for DynamoDB, Kinesis, MQ, apache Kafka, SQS, documentDB
can have up to 5 filters on a single event source mapping. multiple filters are local ORed together by default
Lambda: function states
function states
pending:
function state after function is created
creates resources for the function like VPC resources or EFS resources
any invokatoins during this state will fail
active:
function state after function is created successfully
functions can only be successfully invoked in this state
failed:
resource configuration or provisioning encountered an error
inactive:
the function has been idle long enough for lambda to reclaim the the external resources that were configured for it
on invokation, functions in this state are set to the pending state. If resources are failed to be created after being in this state, the state is set to inactive again
(states of the "LastStatusUpdate" attribute) InProgress: An update is happening on an existing function. While a function update is in progress, invocations go to the function’s previous code and configuration. Successful: The update has completed. Once Lambda finishes the update, this stays set until a further update. Failed: The function update has failed. Lambda aborts the update and the function’s previous code and configuration remain available.
Lambda: errors
errors
invokation error: error occures before function receives invokation request
function error: error occures in the function code or runtime returns an error
when manually invoking a function, Lambda does not automatically retry these types of errors on your behalf. To retry, you can manually re-invoke your function, send the failed event to a queue for debugging, or ignore the error.
Lambda: recursive loop detection
recursive loop detection
allows lambda to detect certain types of recursive loops that invoke your functions in your serverless architecture
can stop the function and notify you
Lambda: function url
lambda function URL
dedicated HTTP(S) endpoint for your Lambda function.
automatically created for you when you create your function
https://<url-id>.lambda-url.<region>.on.aws
can be used to invoke your function
use resource-based policies for security
send a get request to the function to invoke it
can monitor url actions with cloudwatch logs</region></url-id>
Lambda: code signing
code signing
checks each deployment of code to your lambda function and verifies that it comes from a trusted source
not supported for container images
Lambda: tags
tags
free-form key-value pairs useful for implementing ABAC.
applied at the function level, not the alias or version
requires the Lambda:ListTags and Lambda:TagResource permissions
Lambda: access permissions
access permissions
execution role:
defines the permissions that your function needs to access other AWS services and resources.
At a minimum, your function needs access to Amazon CloudWatch Logs for log streaming.
If your function calls other service APIs with the AWS SDK, you must include the necessary permissions in the execution role’s policy.
Lambda also uses the execution role to get permission to read from event sources when you use an event source mapping to invoke your function.
resource based policies are used to determine who can access Lambda resources (functions, layers, aliases, and versions)
if a user tries to access a lambda resource, Lambda considers both the user’s identity-based policies and the resource’s resource-based policy.
When an AWS service calls your Lambda function, Lambda considers only the resource-based policy.
Lambda: ARN format
Lambda ARN format
Function – arn:aws:lambda:us-west-2:123456789012:function:my-function
Function version – arn:aws:lambda:us-west-2:123456789012:function:my-function:1
Function alias – arn:aws:lambda:us-west-2:123456789012:function:my-function:TEST
Event source mapping – arn:aws:lambda:us-west-2:123456789012:event-source-mapping:fa123456-14a1-4fd2-9fec-83de64ad683de6d47
Layer – arn:aws:lambda:us-west-2:123456789012:layer:my-layer
Layer version – arn:aws:lambda:us-west-2:123456789012:layer:my-layer:1
Lambda: data protectoin
Data protection
do not put sensative information onto a tag or free-form text field like a name field
if you provide a url to an external server, do not include credencials information in the url to validate your request to that server
lambda only supports secure connections over https
governance can be acheived using AWS CloudFormation Gaurd and AWS Config
code signing can be configured with AWS Signer
Lambda: function monitoring
function monitoring
lambda is automatically integrated with cloudwatch (as long as the functions execution role has the necessary permissions).
It automatically reports metrics in 1 minute intervals cloudwatch, and shows metrics on the monitoring console
this integration is free, but you can implement custom high-resolution metrics, to which charges apply
error, performance, and concurrency, and asyncronous invokation metrics are all available
Lambda: lambda insights
Lambda Insights
tool to help analyze and identify problems in your lambda microservice architecture
lambda insights uses the cloudwatch lambda insights extension, implemented as a lambda layer.
you are charged for the metrics is collects and sends to cloudwatch
layer content is extracted to the /opt directory of the function
Lambda: extensions, internal vs external
extensions
you are charged for execution of an extension in 1ms increments
internal extensions:
run as part of the runtime process, accessed via wrapper functions or in-process mechanisms
external extensions:
run as a separate process from the execution environment an continuous to run even after the function invokation fully processes
can be written in a language that differs from the runtime, becuase it runs separately from the runtime
Lambda: execution environment, how to speed up execution, execution environment lifecycle phases
execution environment:
the environment in which the lambda function executes.
execution can be sped up by moving any work done out of the function code and into the execution context.
the execution context is re-used between executions
extensions contribute to package size limit
extensions must complete initialization before the function can execute, thereby contibuting to startup time, and they also can impact performance because they share CPU, memory, and storage resources with the function
permissions are also shared between the function and extension
secure and isolated runtime environment that manages function resources and provides lifecycle support lifecycle phases: Init: unfreezes or creates the execution environment, downloads the function code and all layers, initializes extensions, initializes runtime, and runs the functions initialization code (the code outside the main handler) occures during the first invokation (or in advanced if provisioned concurrency is enabled) 3 sub-phases: extension init, runtime init, and function init Restore (SnapStart only): lambda resumes new execution environment from the persisted snapshot instead of initializing a new environment from scratch Invoke: lambda invokes the function handler shutdown: lambda triggers a shutdown if functions haven't been invoked for a long time the runtime is shut down and the environment is removed
Lambda: troubleshooting: general, RequestEntityTooLargeException when calling updateFunction, PermanentRedirect. S3 Error Message: The bucket is in this region…:
troubleshooting
general: permission is denied / cannot load such file:
chmod 755 [filepath] on the code
RequestEntityTooLargeException when calling updateFunction:
zip file size is limited to 50MB when uploading directly to lambda (use S3 bucket to get around this)
PermanentRedirect. S3 Error Message: The bucket is in this region…:
must upload from the same region the function is in
Lambda: rolling deployment
rolling deployment
Use rolling deployments to control the risks associated with introducing new versions of your Lambda function.
In a rolling deployment, the system automatically deploys the new version of the function and gradually sends an increasing amount of traffic to the new version.
Lambda: snapstart
SnapStart
Lambda initializes your function when you publish a function version, then it takes a snapshot of the
memory and disk state of the initialized execution environment, encrypts the snapshot, and caches it for low-latency access
must be a java runtime, and the function must be published
lambda: initialization outside of the handler function
initialization outside of handler function:
initialize the sdk or other reusable assets outside of the handler function.
subsequint invokations can reuse this initialization if it is outside the handler function
SAM: what is it, what do you need to add to a cloudformation template to use it, what types can you add to the resources section of a cloudformation template with SAM, how to push to the cloud with sam (what commands)
Serverless Application Model (SAM)
features templates, an extension of cloudformation, for defining serverless applications
Also use the AWS SAM CLI
*meaning of “Transform: ‘AWS::Serverless-2016-10-31’”: know that this means the cloudformation template is a SAM template
*serverless resources types in ‘Resources’ tag: (AWS::Serverless::Api, AWS::Serverless::Application, AWS::Serverless::Function, AWS::Serverless::HttpApi, AWS::Serverless::LayerVersion, AWS::Serverless::SimpleTable, AWS::Serverless::StateMachine)
‘sam deploy’ zips the underlying cloudformation template and uploads it to S3, and deploys your application to the AWS cloud
sam build builds the application to be ready to deploy with sam deploy
ECR
ECR
fully-managed cloud docker image repo
deploy, store, and manage docker images
Copilot
Copilot
command line tool for deploying, releasing, and operating containerized applications
EKS
EKS
fully-managed kubernetes containerized application orchestration cloud service
ECS
ECS
fully-managed containerized application orchestration cloud service
*what happens to a container instance when it is terminated in a [insert state here] state?
a container that is terminated in a stopped state isn’t automatically removed from the cluster, you must deregister the cluster using the console or CLI if it is to no longer appear as a resource
*/etc/ecs/ecs.config
where ECS configuration variables are stored on Linux variants of the Amazon ECS-optimized AMI
Aurora
Aurora
MySQL and PostgreSQL cloud database service, fully managed and serverless via RDS
features automatic scaling and replications, up to 15 low-latency read replicas, point-in-time recovery, continuous backup to Amazon S3, etc
DynamoDB: what is it?
DynamoDB
cloud key/value document database
DynamoDB: partitioning keys, sort keys, GSIs, LSIs
partitioning keys, LSIs, and GSIs
partitioning key:
primary key: the piece of data that gets sent through a hashing algorithm that determines the partition a piece of data should exist on when being added to the table
sort key: an optional secondary key used to sort the items in their partition
global secondary index (GSIs):
both the partition key and sort key can be different than on the base table
no size limitations
can include data from anywhere in the base table
has its own provisioned throughput settings
local secondary indices (LSIs):
same partition key as base table, different sort key
total size of indexed items for any one partition key value can’t exceed 10 GB
scoped to a base table partition that has the same partition key value
shares provisioned throughput settings for read and write activity with the table it is indexing
cannot have RCUs or WCUs allocated to it directly
DynamoDB: TTL
DynamoDB table TTL
cost-effective method for deleting items that are no longer relevant
define a per-item expiration timestamp that deletes the item when it expires
DynamoDB: PITR vs on-demand backups
on-demand/point in time backup methods, and do you have access to the backups that result?
point in time recovery (PITR):
dynamoDB automatically backs up table data
backups retained for a fixed 35 day period
on-demand backups:
retained forever, even after the table is deleted
DynamoDB: transations
*transactions
dynamoDB transactions allow you to carry out all-or-nothing database operations that can roll back if they fail
DynamoDB: conditional writes
conditional writes:
DynamoDB optionally supports conditional writes for write operations (PutItem, UpdateItem, DeleteItem). A conditional write succeeds only if the item attributes meet one or more expected conditions. Otherwise, it returns an error
DynamoDB: calculating strong RCUs,
calculating WCUs, strongly consistent RCUs, eventually consistent RCUs
Strong RCUs = (s / 4)n, where s = size of data, rounded up to nearest 4. n = number of reads.
Eventual RCUs = round((s / 4)n/2), where s = size of data, rounded up to nearest 4. n = number of reads. round(f) rounds f to the nearest whole
WCU = s*n
DynamoDB: ConsistedRead option
*ConsistentRead=[true, false] option
sets dynamodb to use strongly consistent reads
DynamoDB: streams
DynamoDB Streams:
captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours
Applications can access this log and view the data items as they appeared before and after they were modified, in near-real time
DynamoDB: query vs scan
Query vs scan
Query:
Can only be used if you can create indices
find items in a table based on primary keys. allows you to query a table
filter expression:
determines which items within the Query results should be returned to you.
applied after Query finishes, but before the results are returned.
Scan:
scan through all items and return one or more via filters
parallel scan: scans that run in parallel in independent worker threads, each on their own partition of the table
to be avoided when possible, due to inefficiencies (much less efficient than queries. a large table can use up all your provisioned throughput in a single scan)
DynamoDB: encryption
encryption:
by default, all data in dynamodb is encrypted at rest using KMS
When creating a new table, you can choose one of the following AWS KMS key types to encrypt your table. You can switch between these key types at any time. AWS owned key – Default encryption type. The key is owned by DynamoDB (no additional charge). AWS managed key – The key is stored in your account and is managed by AWS KMS (AWS KMS charges apply). Customer managed key – The key is stored in your account and is created, owned, and managed by you. You have full control over the KMS key (AWS KMS charges apply).
DynamoDB: locking
Locking
Optimistic Locking: no lock is aquired before an entity is accessed. after a modification is made, a check is run to see if the entity has been modified by someone else. if not, the write is completed. else, the modification is abandoned and restarted
Pessimistic Locking: a lock is aquirred to write a resource
Overly Optimistic Locking: it is assumed that no read/write collision will occure, so no effort is made to detect or prevent it
DynamoDB: atomic counters
atomic counters
You can use the UpdateItem operation to implement an atomic counter—a numeric attribute that is incremented, unconditionally, without interfering with other write requests.
can be used to approximate the number of visitors on a website, for example
can be subject to overcounting or undercounting
ElastiCache: What is it?
ElastiCache
web application cache service
supports redis or memcached in-memory caching engines
choosing redis vs memcached for caching
ElastiCache: Memcached vs Redis
memcached:
Is object caching your primary goal, for example to offload your database? If so, use Memcached.
Are you interested in as simple a caching model as possible? If so, use Memcached.
Are you planning on running large cache nodes, and require multithreaded performance with utilization of multiple cores? If so, use Memcached.
Do you want the ability to scale your cache horizontally as you grow? If so, use Memcached.
redis:
Are you looking for more advanced data types, such as lists, hashes, bit arrays, HyperLogLogs, and sets? If so, use Redis.
Does sorting and ranking datasets in memory help you, such as with leaderboards? If so, use Redis.
Are publish and subscribe (pub/sub) capabilities of use to your application? If so, use Redis.
Is persistence of your key store important? If so, use Redis.
Do you want to run in multiple AWS Availability Zones (Multi-AZ) with failover? If so, use Redis.
Is geospatial support important to your applications? If so, use Redis.
Is encryption and compliance to standards, such as PCI DSS, HIPAA, and FedRAMP, required for your business? If so, use Redis.
ElastiCache: use cases
ElastiCache use cases
Your data is slow or expensive to get when compared to cache retrieval.
Users access your data often.
Your data stays relatively the same, or if it changes quickly staleness is not a large issue.
ElastiCache: redis replication, cluster mode enabled/disabled
redis replication: used for read-intensive workloads
cluster mode:
enabled: data partitioned across up to 500 shards
cluster can contain up to 500 shards
all nodes must reside in the same region
you cannot manually promote any nodes to primary
disabled:
cluster contains only a single shard
this shard has one primary read/write node and 0-5 replica nodes
all nodes reside in the same region
ElastiCache: caching strategies
caching strategies:
lazy loading:
your application always requests data from the cache and only loads it into cache from the data store when the data is expired or not present
this avoids filling the cache with data that is never requested
when a cache node fails and is replaced by a new node the application continues to function, it will just have increased latency until it builds back the cache
a cache miss results in 3 trips: original request from the cache, subsiquent request to the data store, and final write to the cache
data is only updated on a cache miss and therefor it can become stale
write-through:
data is added/updated in the cache whenever is is written in the database
cache data is never stale
every write does involve 2 trips (a write to the cache and database) however, users are more likely to tolerate writes taking longer
missing data: new cache nodes have missing data, which continues to be missing uptil its added or updated on the database
cache churn: most data is never read, so writing it to the cache wastes resources
ElastiCache: TTL
TTL:
time to live: set data to expire after a set time to limit data staleness
MemoryDB for redis
MemoryDB for Redis
caching layer for a redis database
RDS: what is it?
RDS
Relational database service in the cloud
supports Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server engines
RDS: read replicas
*read replicas on RDS
A read replica is a read-only copy of a DB instance.
You can reduce the load on your primary DB instance by routing queries from your applications to the read replica.
In this way, you can elastically scale out beyond the capacity constraints of a single DB instance for read-heavy database workloads.
read-only copy of a DB instance allows you to run multiple read-only copies of your database intended to improve performance of your primary (read/write) database must have auto backups enabled up to 5 replicas of a database possible, each with their own DNS endpoint can be multi-AZ or multi-region can have replicas of replicas replicas can be promoted to a master databases (which, necessarily, breaks replication) read replicas do not feature auto failover, a url being used to reference a master must be changed manually to point at a read replica if the master fails make it easy to take advantage of supported engines' built-in replication functionality to elastically scale out beyond the capacity constraints of a single DB instance for read-heavy database workloads. *cross-region read replicas Improve your disaster recovery capabilities. Scale read operations into an AWS Region closer to your users. Make it easier to migrate from a data center in one AWS Region to a data center in another AWS Region.
RDS: automatic backups
*automatic back up feature
RDS has an automated backups feature, enabled by setting the backup retention period setting to a nonzero positive value
your instance is taken offline to conduct a backup
RDS: storage autoscaling
Storage Autoscaling
when Amazon RDS detects that you are running out of free database space it automatically scales up your storage
activates when the following factors apply:
Free available space is less than 10 percent of the allocated storage.
The low-storage condition lasts at least five minutes.
At least six hours have passed since the last storage modification.
amplify
Amplify
manages/provisions mobile app backend
provides a mobile application framework to easily integrate application with IOS, Android, React Native, Web, etc.
automates application release for frontend and backend
cloud9
Cloud9
web-based cloud IDE with terminal, debugger, and editor
CloudShell
CloudShell
web-based cloud shell
CodArtifact
CodeArtifact
cloud software package store
can automatically get latest versions of packages
works with commonly used package managers and build tools like Maven, Gradle, npm, yarn, twine, pip, and NuGet
codebuild: what is it, codebuilt timeouts, bundling dependencies in the build stage vs caching them in S3, getting logs from codebuild to S3, how do you scale codebuild, where does it usually output builds?, how does it roll back deployments, how to encrypt outputs?
CodeBuild
fully managed service that builds code and runs tests
no need to provision a build or test suite
can use default build environments or create your own
*codeBuilt timeouts
if a build is waiting in the queue and doesn’t get picked by by codebuild for a specific timeout value, it is removed from the queue
*bundling things in the source code during the build stage
if deployment is taking a long time due to resolving dependencies, bundle them with the source code during the build stage
if the build stage is taking a long time, cache the dependencies with S3
*Cloudwatch logs/save to S3 integration
you enabled s3 and cloudwatch logs integration to get build logs from codebuild to s3
*know that codebuild scales automatically and that nothing has to be done for scaling or parallel builds
output is usually to S3
rolls back deployments by deploying the latest previous working version of the application
encrypting outputs:
CodeBuild can be pointed to an AWS KMS customer master key to encrypt output
CODEBUILD_KMS_KEY_ID: The identifier of the AWS KMS key that CodeBuild is using to encrypt the build output artifact