Developing Storage Solutions with Amazon S3

Question 1

Differents storage options in AWS

Answer 1

Amazon S3
Amazon S3 Glacier
Amazon Elastic File System (EFS)
Amazon Storage Gateway
Amazon Elastic Block Store (EBS)

Question 2

What is S3 Glacier

Answer 2

Low-cost storage service that provides highly secure, durable, and flexible storage for data archiving and online backup

Question 3

Type of retrieval in S3 Glacier

Answer 3

• Expedited retrievals : 1–5 minutes
• Standard retrievals : 3–5 hours
• Bulk retrievals : 5–12 hours.

Question 4

What is Elastic File System (EFS) and when to use it

Answer 4

Network file system as a service to EC2 instances.

It is designed to meet the performance needs of big data and analytics, media processing, content management, web serving, and home directories.

Question 5

What is Storage Gateway and when to use it

Answer 5

Seamless and secure storage integration between an organization’s on-premises IT environment and the AWS storage infrastructure, such as Amazon S3, Amazon S3 Glacier, and Amazon EBS.

AWS Storage Gateway use cases include the following:
• Corporate file sharing
• Enabling existing on-premises backup applications to store primary backups on Amazon S3
• Disaster recovery
• Mirroring data to cloud-based compute resources and then archiving it to Amazon S3 Glacier

Question 6

What is Elastic Block Store and when to use it

Answer 6

EBS volumes are network-attached storage that persists independently from the running life of a single EC2 instance. With Amazon EBS, you can also create point-in-time snapshots of volumes, which are stored in Amazon S3.

Amazon EBS typical use cases include the following:
• Big data analytics engines (such as the Hadoop/HDFS ecosystem and Amazon EMR clusters)
• Relational and NoSQL databases (such as Microsoft SQL Server and MySQL or Cassandra and MongoDB)
• Stream and log processing applications (such as Kafka and Splunk)
• Data warehousing applications (like Vertica and Teradata)

Question 7

What is Amazon S3 and when to use it

Answer 7

Amazon S3 (simple storage service) provides highly secure, durable, and scalable object storage.

You can use Amazon S3 as a storage solution for use cases such as: 
• Content storage and distribution 
• Backup and archiving 
• Big data analytics 
• Static website hosting 
• Disaster recovery

Question 8

Basic components of Amazon S3

Answer 8

The basic components of Amazon S3 are the bucket, objects, keys, and the unique object url.

Question 9

Different parts of bucket’s URL and object’s URL

Answer 9

https: //[bucket_name].s3.[region endpoint].amazonaws.com
https: //[bucket_name].s3.[region endpoint].amazonaws.com/[object key]

Question 10

Requirements for S3 bucket name

Answer 10

The bucket name must be unique across Amazon S3.

3-63 characters
Lowercase letters, numbers and hyphens (-)
Do not use period (.) which can cause certificate exceptions when accessed with HTTPS
Do not use underscore (_)

A bucket is associated with an AWS Region

Question 11

Requirements for object key name

Answer 11

Encoded in UTF-8
Max 1024 bytes

Safe characters : 0-9 a-z A-Z ! - _ . * ‘ ( ) /

Avoid : \ ; : + = @ , ? & $ ` space % < > [ ] # | { } ^ “ ~ non-printable

Question 12

Two main types of metadata

Answer 12

System-defined metadata includes information such as object creation date, size, and MD5 digest.

User-defined metadata are name-value pairs assigned when an object is uploaded.
The prefix “x-amz-meta-” is automatically added to the metadata name.

Question 13

How does versioning work in S3

Answer 13

An object’s version ID is part of the system-defined metadata.

By default, versioning is disabled in S3 buckets.
• In versioning-disabled buckets, an object has a version ID of null.
• In versioning-enabled buckets, each version of an object has a unique version ID.

Question 14

Old path-style vs. Virtual hosted-style URL

Answer 14

Old path-style :
http://[region specific endpoint]/[bucket name]/[object key]

Virtual hosted-style :
http://[bucket name].s3.amazonaws.com/[object key]

Question 15

Name of operation to upload object and max size of objects uploaded to S3

Answer 15

Upload an object with PUT

You can upload or copy objects of up to 5 GB in a single PUT operation. Larger object => multipart upload

Question 16

How does multipart work

Answer 16

Using multipart upload, you can upload a single object as a set of parts.

You can upload each part separately. If one of the parts fails to upload, you can retransmit that particular part without retransmitting the remaining parts. After all the parts of your object are uploaded to the server, you must send a complete multipart upload request that indicates that multipart upload has been completed. Amazon S3 then assembles these parts and creates the complete object.

You can also stop a multipart upload. When you stop an upload, Amazon S3 deletes all the parts that were already uploaded and frees up storage.

Amazon S3 retains all the parts on the server until you complete or stop the upload. To avoid unnecessary storage costs related to incomplete uploads, complete or stop an upload.

Consider using multipart upload for objects larger than 100 MB

Question 17

The benefits of multipart uploads

Answer 17

• Upload parts in parallel to improve throughput
• Recover quickly from network issues
• Pause and resume object uploads
• Begin an upload before you know the final size of an object

Question 18

When to use copy operations

Answer 18

• Create copies of an object
• Rename an object
• Move it to a different Amazon S3 location
• Update its metadata
• Change the storage class of an object from standard to reduced redundancy or vice versa

Question 19

Name of operation to retrieve an object

Answer 19

GET

You can also retrieve an object in parts by specifying the range of bytes needed.

Question 20

Name of operation to retrieve data within an object

Answer 20

SELECT

Amazon S3 Select analyzes and processes data within an object in Amazon S3 buckets faster and cheaper. You can retrieve a subset of data from an object in Amazon S3 using simple SQL expressions.

Question 21

Name of operation to remove an object

Answer 21

DELETE

You can delete a single object or delete multiple objects in a single delete request.

Versioning disabled
In a bucket that is not versioning-enabled, you can permanently delete an object by specifying the key that you want to delete.

Versioning enabled
In a bucket that is versioning-enabled, you can permanently delete an object by invoking a delete request with a key and version ID. To completely remove an object, you must delete each individual version.

Question 22

How to apply a hierarchy within objects

Answer 22

There is no hierarchy of objects in S3 buckets.

To organize your keys and create a logical hierarchy, you can use delimiters (any string such as / or _) in key names.

If you use prefixes and delimiters to organize keys in a bucket, you can retrieve subsets of keys that match certain criteria. You can list keys by prefix. You can also retrieve a set of common key prefixes by specifying a delimiter.

Question 23

What are presigned URLs, and how to create them

Answer 23

All objects and buckets are private by default. Presigned URLs are useful if you want your user to be able upload a specific object to your bucket without being required to have AWS security credentials or permissions.

A presigned URL is created with:
• Your security credentials
• Bucket name
• Object key
• HTTP method (PUT for uploading objects, GET for retrieving objects)
• Expiration date and time

Share the pre-signed URL with users who need to access your S3 bucket to put or retrieve objects.

Question 24

Different techniques to secure data

Answer 24

SSL-encrypted endpoints with HTTPS
Client-side encryption
Server-side encryption

Question 25

What is SSE-S3

Answer 25

Server-side encryption with Amazon S3-managed keys (SSE-S3)

Each object is encrypted with a unique key employing strong multi-factor encryption. It also encrypts the key itself with a primary key that it regularly rotates. Amazon S3 server-side encryption uses one of the strongest block ciphers available, 256-bit Advanced Encryption Standard (AES-256), to encrypt your data.

Question 26

What is SSE-KMS

Answer 26

Server-side encryption with AWS KMS-managed keys (Key Management Service)

Similar to SSE-S3, with a key management system scaled for the cloud. SSE-KMS also provides you with an audit trail of when your key was used and by whom. Additionally, you can create and manage encryption keys yourself, or use a default key that is unique to you.

Question 27

What is SSE-C

Answer 27

Server-side encryption with customer-provided keys (SSE-C)

You manage the encryption keys. Amazon S3 manages the encryption as it writes to disks and decryption, when you access your objects.

Question 28

When to use ACL on S3

Answer 28

Amazon S3 access control list (ACL) grants coarse-grained user permissions at the bucket or object level. It enables you to apply the principle of least privilege. You can grant users READ, WRITE, or FULL-CONTROL permissions at the object or bucket level.

Question 29

When to use bucket policies on S3

Answer 29

A S3 bucket policy is a IAM policy that grants granular permissions to S3 resources. You add a policy to a bucket to grant other AWS accounts or IAM users access permissions for the bucket and the objects in it.

Question 30

What is CORS for S3

Answer 30

Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in another domain.

Question 31

Best practices for requests on buckets

Answer 31

Avoid unnecessary requests:

• Handle NoSuchBucket errors instead of checking for existence of fixed buckets.
• Set the object metadata before uploading an object.
• Avoid using the copy operation to update metadata.
• Cache bucket and key names if your application design allows it.

Question 32

Best practices for network latency

Answer 32

• Choose the bucket region closest to latency-sensitive customers.
• Compress data stored in Amazon S3.
• Use a CDN, such as Amazon CloudFront, to distribute content, with low-latency and high data-transfer speeds.

Question 33

Best practices for data integrity

Answer 33

• Ensure that data has not been corrupted in transit.

- Check the MD5 checksum of the object retrieved from the GET and PUT operation.

Question 34

Differences between 400 and 500 error codes

Answer 34

A 400-series error code indicates that you cannot perform the operation.

• BucketAlreadyExists
• InvalidBucketName

A 500-series error code indicates that you may retry the operation.
- InternalError
- SlowDown
With an exponential backoff algorithm, you specify progressively longer waits after each failed attempt before retrying your request