Database Specialty - CloudFormation and Automation Flashcards
Infrastructure as Code
- Currently, we have been doing a lot of manual work
- All this manual work will be very tough to reproduce:
- In another region
- in another AWS account
- Within the same region if everything was deleted
- Wouldn’t it be great, if all our infrastructure was… code?
- That code would be deployed and create / update / delete our infrastructure
What is CloudFormation
- CloudFormation is a declarative way of outlining your AWS Infrastructure, for
any resources (most of them are supported). - For example, within a CloudFormation template, you say:
- I want a security group
- I want two EC2 machines using this security group
- I want two Elastic IPs for these EC2 machines
- I want an S3 bucket
- I want a load balancer (ELB) in front of these machines
- Then CloudFormation creates those for you, in the right order, with the exact
configuration that you specify
Benefits of AWS CloudFormation (1/2)
- Infrastructure as code
- No resources are manually created, which is excellent for control
- The code can be version controlled for example using git
- Changes to the infrastructure are reviewed through code
- Cost
- Each resources within the stack is tagged with an identifier so you can easily see how
much a stack costs you - You can estimate the costs of your resources using the CloudFormation template
- Savings strategy: In Dev, you could automation deletion of templates at 5 PM and
recreated at 8 AM, safely
- Each resources within the stack is tagged with an identifier so you can easily see how
Benefits of AWS CloudFormation (2/2)
- Productivity
- Ability to destroy and re-create an infrastructure on the cloud on the fly
- Automated generation of Diagram for your templates!
- Declarative programming (no need to figure out ordering and orchestration)
- Separation of concern: create many stacks for many apps, and many layers. Ex:
- VPC stacks
- Network stacks
- App stacks
- Don’t re-invent the wheel
- Leverage existing templates on the web!
- Leverage the documentation
How CloudFormation Works
- Templates have to be uploaded in S3 and then referenced in CloudFormation
- To update a template, we can’t edit previous ones. We have to reupload a new version
of the template to AWS - Stacks are identified by a name
- Deleting a stack deletes every single artifact that was created by
CloudFormation.
Deploying CloudFormation templates
- Manual way:
- Editing templates in the CloudFormation Designer
- Using the console to input parameters, etc
- Automated way:
- Editing templates in a YAML file
- Using the AWS CLI (Command Line Interface) to deploy the templates
- Recommended way when you fully want to automate your flow
CloudFormation Building Blocks
Templates components (one course section for each):
1. Resources: your AWS resources declared in the template (MANDATORY)
2. Parameters: the dynamic inputs for your template
3. Mappings: the static variables for your template
4. Outputs: References to what has been created
5. Conditionals: List of conditions to perform resource creation
6. Metadata
Templates helpers:
1. References
2. Functions
YAML Crash Course
- YAML and JSON are the languages you can
use for CloudFormation. - JSON is horrible for CF
- YAML is great in so many ways
- Let’s learn a bit about it!
- Key value Pairs
- Nested objects
- Support Arrays
- Multi line strings
- Can include comments
What are resources?
- Resources are the core of your CloudFormation template (MANDATORY)
- They represent the different AWS Components that will be created and
configured - Resources are declared and can reference each other
- AWS figures out creation, updates and deletes of resources for us
- There are over 224 types of resources (!)
- Resource types identifiers are of the form:
AWS::aws-product-name::data-type-name
How do I find resources documentation?
- I can’t teach you all of the 224 resources, but I can teach you how to
learn how to use them. - All the resources can be found here:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aw
s-template-resource-type-ref.html - Then, we just read the docs J
- Example here (for an EC2 instance):
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aw
s-properties-ec2-instance.html
Analysis of CloudFormation Template
- Going back to the example of the introductory section, let’s learn why it
was written this way. - Relevant documentation can be found here:
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/awsproperties-ec2-instance.html
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/awsproperties-ec2-security-group.html
- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/awsproperties-ec2-eip.htm
What are parameters?
- Parameters are a way to provide inputs to your AWS CloudFormation
template - They’re important to know about if:
- You want to reuse your templates across the company
- Some inputs can not be determined ahead of time
- Parameters are extremely powerful, controlled, and can prevent errors
from happening in your templates thanks to types.
When should you use a parameter?
- Ask yourself this:
- Is this CloudFormation resource configuration likely to change in the future?
- If so, make it a parameter.
- You won’t have to re-upload a template to change its content
How to Reference a Parameter
- The Fn::Ref function can be leveraged to reference parameters
- Parameters can be used anywhere in a template.
- The shorthand for this in YAML is !Ref
- The function can also reference other elements within the template
Concept: Pseudo Parameters
- AWS offers us pseudo parameters in any CloudFormation template.
- These can be used at any time and are enabled by default
What are mappings?
- Mappings are fixed variables within your CloudFormation Template.
- They’re very handy to differentiate between different environments
(dev vs prod), regions (AWS regions), AMI types, etc - All the values are hardcoded within the template
When would you use mappings vs parameters ?
- Mappings are great when you know in advance all the values that can be
taken and that they can be deduced from variables such as- Region
- Availability Zone
- AWS Account
- Environment (dev vs prod)
- Etc…
- They allow safer control over the template.
- Use parameters when the values are really user specific
Fn::FindInMap
Accessing Mapping Values
- We use Fn::FindInMap to return a named value from a specific key
- !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]
What are outputs?
- The Outputs section declares optional outputs values that we can import into
other stacks (if you export them first)! - You can also view the outputs in the AWS Console or in using the AWS CLI
- They’re very useful for example if you define a network CloudFormation, and
output the variables such as VPC ID and your Subnet IDs - It’s the best way to perform some collaboration cross stack, as you let expert
handle their own part of the stack - You can’t delete a CloudFormation Stack if its outputs are being referenced
by another CloudFormation stack
Outputs Example
- Creating a SSH Security Group as part of one template
- We create an output that references that security group
Cross Stack Reference
- We then create a second template that leverages that security group
- For this, we use the Fn::ImportValue function
- You can’t delete the underlying stack until all the references are deleted
too.
What are conditions used for?
- Conditions are used to control the creation of resources or outputs
based on a condition. - Conditions can be whatever you want them to be, but common ones
are:- Environment (dev / test / prod)
- AWS Region
- Any parameter value
- Each condition can reference another condition, parameter value or
mapping
How to define a condition?
- The logical ID is for you to choose. It’s how you name condition
- The intrinsic function (logical) can be any of the following:
- Fn::And
- Fn::Equals
- Fn::If
- Fn::Not
- Fn::Or
Using a Condition
- Conditions can be applied to resources / outputs / etc…
Resources:
MountPoint:
Type: “AWS: :EC2: :VolumeAttachment”
Condition: CreateProdResources
CloudFormation
Must Know Intrinsic Functions
- Ref
- Fn::GetAtt
- Fn::FindInMap
- Fn::ImportValue
- Fn::Join
- Fn::Sub
- Condition Functions (Fn::If, Fn::Not, Fn::Equals, etc…)
Fn::Ref
- The Fn::Ref function can be leveraged to reference
- Parameters => returns the value of the parameter
- Resources => returns the physical ID of the underlying resource (ex: EC2 ID)
- The shorthand for this in YAML is !Ref
DbSubnet1:
Type: AWS: :EC2: :Subnet
Properties:
VpcId: !Ref MyVPC
Fn::GetAtt
- Attributes are attached to any resources you create
- To know the attributes of your resources, the best place to look at is
the documentation. - For example: the AZ of an EC2 machine!
Fn::FindInMap
Accessing Mapping Values
- We use Fn::FindInMap to return a named value from a specific key
- !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]
Fn::ImportValue
- Import values that are exported in other templates * For this, we use the
Fn::ImportValue function
Fn::Join
- Join values with a delimiter
- This creates “a:b:c”
Function Fn::Sub
- Fn::Sub, or !Sub as a shorthand, is used to substitute variables from a
text. It’s a very handy function that will allow you to fully customize your
templates. - For example, you can combine Fn::Sub with References or AWS Pseudo
variables!
Condition Functions
- The logical ID is for you to choose. It’s how you name condition
- The intrinsic function (logical) can be any of the following:
- Fn::And
- Fn::Equals
- Fn::If
- Fn::Not
- Fn::Or
Conditions:
CreateProdResources: !Equals [ !Ref EnvType, prod]
CloudFormation Rollbacks
- Stack Creation Fails:
- Default: everything rolls back (gets deleted). We can look at the log
- Option to disable rollback and troubleshoot what happened
- Stack Update Fails:
- The stack automatically rolls back to the previous known working state
- Ability to see in the log what happened and error messages
ChangeSets
- When you update a stack, you need to know what changes before it
happens for greater confidence - ChangeSets won’t say if the update will be successful
Nested stacks
- Nested stacks are stacks as part of other stacks
- They allow you to isolate repeated patterns / common components in
separate stacks and call them from other stacks - Example:
- Load Balancer configuration that is re-used
- Security Group that is re-used
- Nested stacks are considered best practice
- To update a nested stack, always update the parent (root stack)
CloudFormation – Cross Stacks
- Helpful when stacks have different lifecycles
- Use Outputs Export and Fn::ImportValue
- When you need to pass export values to
many stacks (VPC Id, etc…)
CloudFormation – Nested Stacks
- Helpful when components must be re-used
- Ex: re-use how to properly configure an
Application Load Balancer - The nested stack only is important to the
higher level stack (it’s not shared)
CloudFormation - StackSets
- Create, update, or delete stacks across multiple accounts and regions with a
single operation - Administrator account to create StackSets
- Trusted accounts to create, update, delete stack instances from StackSets
- When you update a stack set, all associated stack instances are updated throughout all accounts and regions.
CloudFormation for DB Specialty Exam
- AWS::RDS::DBInstance
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws- properties-rds-database-instance.html - Important Parameters:
- DeleteAutomatedBackups: remove automated backups immediately after the DB
instance is deleted. Default value: remove all automated backups - DeletionProtection: enable deletion protection on the DB in RDS. The database can’t
be deleted while it has DeletionProtection on. - MasterUsername / MasterPassword: must be provided as text values (we’ll see how
to handle that securely later on) - EnableIAMDatabaseAuthentication: can be enabled, does not replace Master
username & password - DBClusterIdentifier: (optional) if the DB belongs to an Aurora DB cluster
- DeleteAutomatedBackups: remove automated backups immediately after the DB
RDS and CloudFormation – DB Updates
- If Update Requires: Replacement:
- New DB is created
- Links are updated
- Old DB is deleted
- If a DB instance is deleted or replaced during an update, AWS
CloudFormation deletes all automated snapshots. - However, it retains manual DB snapshots
- During an update that requires replacement, you can apply a stack
policy to prevent DB instances from being replaced
RDS and CloudFormation – database update
- Take a snapshot before updating an update. If you don’t, you lose
the data when AWS CloudFormation replaces your DB instance.
To preserve your data: - Deactivate any applications that are using the DB instance so that there’s
no activity on the DB instance. - Create a snapshot of the DB instance
- If you want to restore your instance using a DB snapshot, modify the
updated template with your DB instance changes and add
the DBSnapshotIdentifier property with the ID of the DB snapshot that
you want to use. - After you restore a DB instance with a DBSnapshotIdentifier property,
you must specify the same DBSnapshotIdentifier property for any future
updates to the DB instance. - Update the stack.
- DBSnapshotIdentifier is also useful for mass creating dev/test
databases with some data from prod
CloudFormation protections
- CloudFormation Stack Termination Protection: can’t delete the stack without removing it (but you can still update the stack)
- CloudFormation Stack Policy: prevent updates to CloudFormation templates.
- CloudFormation DeletionPolicy: what happens if a resource gets deleted
* Delete: delete the resource (default behaviour for most resources)
* Retain: do not delete the resources
* Snapshot: (only to resources that support snapshot) snapshot then delete - The default CloudFormation behavior for RDS depends on DBClusterIdentifier :
- If DBClusterIdentifier not specified (= non Aurora), CloudFormation will snapshot
- If DBClusterIdentifier is specified (=Aurora), CloudFormation deletes the DB instance
Aurora with CloudFormation
- First create an AWS::RDS::DBCluster
- Then add AWS::RDS::DBInstance
- Default DeletionPolicy of
RDS::DBCluster is Snapshot
* Means if individual DBInstance are deleted, no snapshot is made (default)
* If the DBCluster is deleted, a snapshot will be made - RDS::DBInstance will inherit
configurations
CloudFormation Parameters
- Parameters are referenced using !Ref
- They can have 4 different sources:
* (static) Given using the CLI / Management console
* (static) Given using a flat file from local filesystem or S3
* (dynamic) From the SSM Parameter Store
* (dynamic) From AWS Secrets Manager - The last two are the most secure way
- Exam trap: you cannot pass in KMS encrypted parameters through
* Use SSM Parameter Store or Secrets Manager
instead!
CloudFormation Parameters - SSM
- Retrieve a String (plaintext) or SecureString
(KMS encrypted) from SSM Parameter Store
at runtime!- Amazing to store environment specific variables
- Hierarchical way of storing parameters
- SSM CF parameters: will be resolved every time you run the template, does not work for SecureString.
- Dynamic parameter pattern:
‘{{resolve:ssm:parameter-name:version}}’, you
must specify the exact version, works for
SecureString.
CF Parameters – Secrets Manager
- Secrets rotation + integration with RDS =
- Dynamic parameter pattern:
‘{{resolve:ssm:parameter-name:version}}’, you must
specify the exact version - Neither Secrets Manager nor CloudFormation logs
or persists any resolved secret value - Updating a secret in Secrets Manager does not
automatically update the secret in CloudFormation - To manage updating the secret in your template,
consider using version-id to specify the version of
your secret. - {{resolve:secretsmanager:secret-id:secret- string:json-key:version-stage:version-id}
Secrets Manager + RDS
- Create a AWS::SecretsManager::Secret
* Specify GenerateSecretString to randomly generate a secret - Create an AWS::RDS::DBInstance that references that secret
* Using dynamic references - Create a
AWS::SecretsManager::SecretTargetAttachment- Link to Secrets Manager secret and the RDS database together
- Will add RDS properties to Secret Manager JSON
Secrets Manager Rotation Schedule
- Create a
AWS::SecretsManager::RotationSchedule
* It will create a Lambda
function
* You must do this after
creating the secret
attachment
Other CloudFormation concepts
- CloudFormation Drift:
- See if the resource configurations have changed from the template over time
- Stack Set:
- Apply the same CloudFormation template into many accounts / regions
- Change Set:
- Evaluate the changes that will be applied before applying them (safety)