Advanced EC2 Flashcards
Bootstrapping EC2 using User Data
Bootstrapping is a process where scripts or other bits of configuration can be run, when an instance is first launched, meaning that an instance, can be brought into service in a certain pre-configured state.
-Allows EC2 Build Automation
-User Data - Accessed via the meta-data IP
-http://169.254.169.254/latest/user-data
-User Data - Is a piece of data that you can pass into an EC2 instance
-Anything in User Data is executed by the instance OS (ONLY AT LAUNCH)
-EC2 doesn’t interpret, the OS needs to understand the User Data
User Data - Key Points
-It’s opaque to EC2 - It’s just a block of data
-It’s NOT secure - don’t use it for passwords or long term credentials (Anyone with access to the O.S can see the User Data)
-Limited to 16 KB in size
-Can be modified when instance stopped
-Only executed one at launch
Boot-Time-To-Service-Time
How quickly you can bring an instance to a service.
AMI > minutes > Instance
-Post Launch Time = Time required after launch, for you to perform manual configuration or automatic configuration before the instance is ready for service
-You can do it quickly using Bootstrapping
-You can Bake an AMI
The optimal way is to combine both of these processes: AMI Baking and Bootstrapping
-You’d use AMI baking for any part of the process, which is time intensive, for example, if you have an application which is 90% installation (AMI baking) and 10% configuration (Bootstrapping).
That way, you reduce the post launch time and thus the boot-time-to-service-time.
Enhanced Bootstrapping with CFN-INIT (AWS::CloudFormation::Init)
Is a way that you can pass complex bootstrapping instructions into an EC2 instance.
-cfn-init helper script - installed on EC2 OS
-Simple configuration management system
-Procedural (User Data) - Run by the O.S line by line
-Desired State (cfn-init)
Where you direct it how you want something to be, it will perform whatever is required to move the instance into that desired state. For example, you can tell cfn-init, that you want a certain version of the Apache web server to be installed and, if that’s already the case, and it’s the same version then nothing is done. If Apache is not installed, then cfn-init will install it or it will update any older versions to that version.
-It can make sure Packages are installed,
-It can manipulate O.S Groups and Users,
-It can download Sources and extract them onto the local instance, even using authentication,
-It can create files with certain contents, permissions and ownerships
-It can run Commands and test that certain conditions are true, after the commands are run
-It can control Services on an instance, ensuring that a particular service is started or enabled.
-Provided with directives via Metadata and AWS::CloudFormation::Init on a CFN resource
-It can be configured to watch for updates to the metadata, on a object in a template. If that metadata changes, then cfn-init can be executed again and it will update the configuration of that instance to the desired state specified inside the template
CreationPolicy and Signals
A CreationPolicy is something which is added to a logical resource inside a CF template. You create it and you supply a timeout value.
This is used to create a stack which creates an instance, at this point CF waits, it doesn’t move the instance into a create complete status when EC2 signals, that it has been created successfully. Instead, it waits for a signal, a signal from the resource itself.
-The cfn-signal command understands, how to communicate with the specific CF stack that it’s running inside
-If the output of the “desired state configuration command” is an okay state, then the okay is sent as a signal by cfn-signal
-If cfn-init reports an error code, then this is sent using cfn-signal to the CF stack
-So cfn-signal is reporting to CF, the success or not of the cfn-init bootstrapping and this is reported to the CF stack.
-If it’s success code, so if cfn-init worked as intended, then the resource is moved to a “Create Complete State”
-If cfn-signal reports an error, the resourcing CF shows an error
-If nothing happens for the time period that you specified on the CF stack, then CF assumes, it’s errored and doesn’t let the stack create successfully
EC2 Instance Roles
Allowing a service to assume a role, grants the service the permisions that the role has.
EC2 instance roles are roles that an instance can assume, and anything running in that instance has the permissions that the role grants.
Architecture
IAM Role with a Permissions Policy attached - So whoever assumes the role gets temporary credentials generated, and those give the permissions, that the policy has.
-Means that an EC2 instance itself can assume it and gain access to those credentials. So that the application running inside that instance, can use the permissions that the role provides.
There’s an intermediate piece of architecture, called the “Instance Profile”, and this is a wrapper around an IAM role. This is the thing, that allows the permissions to get inside the instance.
-When you create an Instance Role, an Instance Profile, is created with the same name.
-If you use the command line or CF, you need to create these two things separately
-When using the UI, and you think you’re attaching an instance role direct to an instance, YOU’RE NOT, you are attaching an instance profile of the same name. It’s the Instance Profile that’s attached to an EC2 instance.
-Inside an EC2 instance, temporary credentials are delivered via the Instance Metadata **
-EC2 and the secure token service liaise with each other, to ensure that the credentials are always renewed, before they expire
EC2 Instance Roles - Summary
-Credentials are inside meta-data
-Inside the meta-data, there’s an IAM tree = iam/security-credentials/role-name
-Automatically rotated - Always valid
-Should always be used rather than adding access keys into instance
-CLI tools will use ROLE credentials automatically
AWS Systems Manager Parameter Store (SSM Parameter Store)
A service from AWS, which makes it easy to store various bits of systems configuration (strings, documents, secrets), and store those in a resilient, secure, and scalable way.
Remember that passing secrets into an EC2 instance using UserData was bad practice, because anyone with access to the instance could access all of that data. Parameter Store is a way that this can be improved.
-Lets you create parameters (they have a name and a value, this is the part that stores the actual configuration)
-Storage for configuration & secrets
-Offers the ability to store 3 different types of parameters: String, StringList & SecureString
-Using these different types of parameters, you can store things inside the product such as: Lincense codes, Database Strings, Full Configs & Passwords.
-Allows you to store parameters using a Hierarchical structure & stores different versions of parameters (Versioning)
-Allows you to store Plaintext and Ciphertext (Integrates with KMS to encrypt parameters)
-Public Parameters - Uses Latest AMIs per region
SSM Parameter Store - Architecture
-Different types of things can use the Parameter Store: EC2 Instances (and all things inside of it), Applications and Lambda Functions
-They can all request access to parameters inside the Parameter Store
-Parameter Store is integrated with IAM for permissions, so any accesses will need to be authenticated and authorized and that might use long-term credentials (access keys) or short-term passed in via an IAM role
-If parameters are encrypted then KMS will be involved and the apropiate permissions to the CMK inside KMS will also be required
-Parameter Store allows you to create a simple (myDBpassword) or complex sets of parameters (Hierarchical structure, such as /Wordpress/ that has a DBUser, which could be accessed either by using its full name or requesting the Wordpress part of the tree )
-Any changes that occur to any parameters, can create events, and these events can also start processes in other AWS services.
System and Application Logging on EC2
-CloudWatch is for metrics
-ClouwWatch Logs is for logging
-Neither natively capture data inside an Insatance
-CloudWatch Agent is required
-To function, it needs to have the configuration and permissions
CloudWatch Agent - Architecture
-We need to supply a configuration to the Agent, so that he knows what to do
-The Agent also needs someway of interacting with AWS on permissions - IAM Role with permissions to interact with CW Logs
-The Agent Config also configures the metrics and the logs, we want to capture, and these are all injected into CW using Log Groups.
-We’l configure one log group for every log file, that we want to inject into the product and then within each Log Group, there’ll be a Log Stream, for each instance performing this logging.
-You can install it manually or use CF (automation) to include that Agent configuration for every instance that you provision
-CW Agent comes with a number of ways to obtain and store the configuration, that it will use to send this data into CW Logs
One of those ways is we can actually use the Parameter Store and store the Agent configuration as a parameter.
EC2 Placement Groups
When you launch an EC2 instance, it’s physical location is selected by AWS, placing it on whatever EC2 host makes the most sense within the AZ, that’s launched in.
-Placement Groups allow you to influence placement - Ensuring that instances are either physically close together or not
There are currently 3 types of placement groups for EC2:
All of them influence how instances are arranged on physical hardware, but each of them do it for different underlying reasons.
-Cluster placement - Ensure that any instances in a single cluster placement group are physically close together. (Pack instances close together)
-Spread placement - Ensure that instances are all using different underlying hardware (Keep instances separated)
-Partition - Designed for distributed and replicated applications, which have infrastructure awareness. So where you want groups of instances but where each group is different hardware. (Group of instances spread apart)
Cluster Placement Groups
-Are used where you want to achieve the absolute highest level of performance possible within EC2
-With Cluster, you create a group and best practice is that you launch all of the instances, in this group at the same time - This ensures that AWS allocate capacity for everything that you require
-They have to be launched in a single AZ
-When you create a placement group, you don’t specify an AZ, instead when you launch the first instance or instances, into that placement group, it will lock that placement group to whichever AZ that instance it’s also launched into.
-Generally use the same Rack, but often the same EC2 Host
-All the instances within a placement group, have fast direct bandwidth to all other instances inside it. (All members have direct connections to each other)
-When transferring data between instances within that Cluster placement group, they can achieve single stream transfer rates of 10Gbps p/ stream vs the usual 5GB per second
-Because of the physical placement, they’re the Lowest Latency Possible and the maximum packets per second (PPS) possible in AWS. (To achive these levels of performance, you’ll need instances with high performance networking) (also use Enhanced Networking)
Cluster Placement Groups are used when you REALLY need performance, that needed to achieve the highest levels of thoughput and the lowest consistent latencies within AWS.
But the trade-off is because of the physical location, if the hardware that they’re running on fails, logically it could take down all the instances within that cluster placement group.
-Cluster Placement Groups offer little to no resilience
Cluster Placement Groups - KEY POINTS
-Can’t span AZs - ONE AZ ONLY - locked when launching first instance
-Can span VPC peers - but impacts performance
-Requires a supported instance type
-Use the same type of instance (not mandatory)
-Launch at the same time (not mandatory… very recommended)
-10Gbps single stream performance
-Use case: Performance, fast speeds, low latency
Spread Placement Groups
-Designed to ensure the maximum amount of availability and resilience for an application
-Can span multiple AZs
-Instances which are placed into a spread placement group, are located on separate isolated infrastructure racks, within each AZ.
-Each instance has its own isolated networking and power supply, separate from any of the other instances, also within that same spread placement group. (if a single rack fails, either from a networking or a power perspective, the fault can be isolated to one of those racks)
-Limit of 7 Instances per AZ - Because each instance is in a completely separate infrastructure rack and there are limits on the number of these within each AZ
