Storage | AWS Storage Gateway Flashcards
What is the AWS Storage Gateway service?
General
AWS Storage Gateway | Storage
The AWS Storage Gateway service enables hybrid storage between on-premises environments and the AWS Cloud. It seamlessly integrates on-premises enterprise applications and workflows with Amazon’s block and object cloud storage services through industry standard storage protocols. It provides low-latency performance by caching frequently accessed data on premises, while storing data securely and durably in Amazon cloud storage services. It provides an optimized data transfer mechanism and bandwidth management, which tolerates unreliable networks and minimizes the amount of data being transferred. It brings the security, manageability, durability, and scalability of AWS to existing enterprise environments through native integration with AWS encryption, identity management, monitoring, and storage services. Typical use cases include backup and archiving, disaster recovery, moving data to S3 for in-cloud workloads, and tiered storage.
AWS Storage Gateway supports three storage interfaces: file, volume, and tape. Each gateway you have can provide one type of interface.
The file gateway enables you to store and retrieve objects in Amazon S3 using file protocols, such as NFS. Objects written through file gateway can be directly accessed in S3.
The volume gateway provides block storage to your applications using the iSCSI protocol. Data on the volumes is stored in Amazon S3. To access your iSCSI volumes in AWS, you can take EBS snapshots which can be used to create EBS volumes.
The tape gateway provides your backup application with an iSCSI virtual tape library (VTL) interface, consisting of a virtual media changer, virtual tape drives, and virtual tapes. Virtual tape data is stored in Amazon S3 or can be archived to Amazon Glacier.
How do I use the AWS Storage Gateway service?
General
AWS Storage Gateway | Storage
You can have two touchpoints to use the service: the AWS Management Console and a gateway virtual machine (VM).
You use the AWS Management Console to download the gateway, configure storage, and manage and monitor the service. The gateway connects your applications to AWS storage by providing standard storage interfaces. It provides transparent caching, efficient data transfer, and integration with AWS monitoring and security services.
To get started, sign up for the AWS Storage Gateway by choosing “Sign Up Now” on the AWS Storage Gateway detail page. To sign-up, you must have an Amazon Web Services account; if you don’t already have one, you are prompted to create one when you begin the AWS Storage Gateway sign-up process.
After you sign up, you visit the AWS Storage Gateway Management Console to download a gateway with a file, volume, or tape interface. Once you’ve downloaded and installed your gateway, you associate it with your AWS Account through our activation process. After activation, you configure the gateway to connect to the appropriate storage type. For file gateway, you configure file shares that are mapped to selected S3 buckets, using IAM roles. For volume gateway, you create and mount volumes as iSCSI devices. For tape gateway, you connect your backup application to create and manage tapes. Once configured, you start using the gateway to write and read data to and from AWS storage. You can monitor the status of your data transfer and your storage interfaces through the AWS Management Console. Additionally, you can use the API or SDK to programmatically manage your application’s interaction with the gateway.
What is file gateway?
General
AWS Storage Gateway | Storage
File gateway provides a virtual file server, which enables you to store and retrieve Amazon S3 objects through standard file storage protocols. File gateway allows your existing file-based applications or devices to use secure and durable cloud storage without needing to be modified. With file gateway, your configured S3 buckets will be available as Network File System (NFS) mount points. Your applications read and write files and directories over NFS, interfacing to the gateway as a file server. In turn, the gateway translates these file operations into object requests on your S3 buckets. Your most recently used data is cached on the gateway for low-latency access, and data transfer between your data center and AWS is fully managed and optimized by the gateway. Once in S3, you can access the objects directly or manage them using features such as S3 Lifecycle Policies, object versioning, and cross-region replication. You can run file gateway on-premises or in EC2.
What is volume gateway?
General
AWS Storage Gateway | Storage
Volume gateway provides an iSCSI target, which enables you to create volumes and mount them as iSCSI devices from your on-premises or EC2 application servers. The volume gateway runs in either a cached or stored mode.
In the cached mode, your primary data is written to S3, while retaining your frequently accessed data locally in a cache for low-latency access.
In the stored mode, your primary data is stored locally and your entire dataset is available for low-latency access while asynchronously backed up to AWS.
In either mode, you can take point-in-time snapshots of your volumes and store them in Amazon S3, enabling you to make space-efficient versioned copies of your volumes for data protection and various data reuse needs.
What is tape gateway?
General
AWS Storage Gateway | Storage
Tape gateway presents your backup application with a virtual tape library (VTL) interface, consisting of a media changer and tape drives. You can create virtual tapes in your virtual tape library using the AWS Management Console. Your backup application can read data from or write data to virtual tapes by mounting them to virtual tape drives using the virtual media changer. Virtual tapes are discovered by your backup application using its standard media inventory procedure. Virtual tapes are available for immediate access and are backed by Amazon S3. You can also archive tapes. Archived tapes are stored in Amazon Glacier.
What benefits does AWS Storage Gateway provide?
General
AWS Storage Gateway | Storage
AWS Storage Gateway provides a set of features that enable you to effectively leverage AWS storage within your existing applications and workflows. It provides a standard set of protocols such as iSCSI and NFS, which allow you to use your existing applications without any changes. Through its local cache, the gateway provides low-latency access to frequently used data. The gateway optimizes data transfer to AWS storage, such as optimization of transfer through intelligent buffering, upload management to address network variations, and bandwidth management. The gateway provides you an effective mechanism to store data in AWS across the range of storage services most suitable for your use cases. The gateway is easy to deploy and leverages your existing virtual infrastructure investments and integrates with native hypervisors. The gateway is stateless, allowing you to easily create and manage new instances of your gateway as your storage needs evolve. Finally, it integrates natively into AWS management services such as Amazon CloudWatch, AWS CloudTrail, AWS KMS, and IAM.
What sort of encryption does AWS Storage Gateway use to protect my data?
General
AWS Storage Gateway | Storage
All data transferred between any type of gateway appliance and AWS storage is encrypted using SSL. By default, all data stored by AWS Storage Gateway in S3 is encrypted server-side with Amazon S3-Managed Encryption Keys (SSE-S3). Also, when using the file gateway, you can optionally configure each file share to have your objects encrypted with AWS KMS-Managed Keys using SSE-KMS.
Is AWS Storage Gateway HIPAA eligible?
File Gateway
AWS Storage Gateway | Storage
Yes. AWS Storage Gateway is HIPAA eligible. If you have an executed Business Associate Agreement (BAA) with AWS, you can use Storage Gateway to store, backup and archive protected health information (PHI) on scalable, cost-effective, and secure AWS storage services, including Amazon S3, Amazon Glacier and Amazon EBS, which are also HIPAA eligible.
Information on HIPAA eligible services on AWS can be found at our HIPAA Compliance page, and you can enter into a BAA with AWS here. HIPAA eligibility for Storage Gateway applies to all gateway types (File, Volume and Tape).
What is file gateway?
File Gateway
AWS Storage Gateway | Storage
File gateway provides a virtual on-premises file server, which enables you to store and retrieve Amazon S3 objects through standard file storage protocols.
What can I do with file gateway?
File Gateway
AWS Storage Gateway | Storage
The use cases for file gateway include: (a) ingesting file-based data into S3 for object workloads, (b) “cloud-bursting”, where you can move data to AWS, run a workload, and access results within your on-premises application, (c) migrating file to objects in S3, (d) cost-effective storage for backup and archive, (f) disaster recovery, and (g) tiering on-premises file storage to S3.
What are the benefits of using file gateway to store data in S3?
File Gateway
AWS Storage Gateway | Storage
File gateway enables your existing file-based applications, devices, and workflows to use cloud storage without modification. File gateway securely and durably stores both file contents and metadata as objects in your Amazon S3 buckets using standard file protocols.
What protocols are supported? Can I use Microsoft Windows?
File Gateway
AWS Storage Gateway | Storage
File gateway supports clients connecting to the gateway using NFS v3 and v4.1. Microsoft Windows clients that support NFS v3 can connect to file gateway.
How do I create and configure a file share?
File Gateway
AWS Storage Gateway | Storage
You can create a file share using the AWS Management Console or service API, and associate it with a new or existing S3 bucket. You can configure your file share with administative controls such as limiting acces to specific NFS clients or networks, read-only or read-write, or enabling user permisssion squashing.
How does file gateway access my S3 bucket?
File Gateway
AWS Storage Gateway | Storage
File gateway uses an AWS Identity and Access Management (IAM) role to access your S3 bucket. You can set this up yourself, or have it automatically setup by the AWS Storage Gateway Management Console. For automatic setup, AWS Storage Gateway will create a new IAM role in your account and associate it with an IAM Access Policy to access your S3 bucket. The IAM role and IAM access policy are created in your account and you can fully manage them yourself.
How does my application access my file share?
File Gateway
AWS Storage Gateway | Storage
To use the file share, you mount it from your application using standard UNIX or Windows commands. For convenience, example command lines are shown in the management console
What is the relationship between files and objects?
File Gateway
AWS Storage Gateway | Storage
Files are stored as objects in your S3 buckets and you can configure the initial storage class for objects that file gateway creates. There is a one-to-one relationship between files and objects, and you can configure the initial storage class for objects that file gateway creates.
The object key is derived from the file path within the file system. For example, if you have a gateway with hostname file.amazon.com and have mapped my-bucket, then file gateway will expose a mount point called file.amazon.com:/export/my-bucket. If you then mount this locally on /mnt/my-bucket and create a file named file.html in a directory /mnt/my-bucket/dir this file will be stored as an object in the bucket my-bucket with a key of dir/file.html.
What file system operations are supported by file gateway?
File Gateway
AWS Storage Gateway | Storage
Your NFS client can create, read, update, and delete, files and directories. Files are stored as individual objects in S3. Directories are managed as folder objects in S3, using the same syntax as the S3 console.
Symbolic links and hard links are not supported. Attempting to create a link will result in an NFS error. Creating sparse files will result in a non-sparse zero-filled object in S3.
Rename operations will appear atomic to NFS clients, but S3 does not support renaming of objects. When you rename a file or directory the gateway performs copy-put requests to create a copy of the objects in S3 under the new keys and then deletes the originals. This avoids having to re-send large files over the network. Renaming directories containing a large number of files is not instantaneous and will result in 2 copies of your data being stored in S3 until the rename operation completes.
What file system metadata can my NFS client access?
File Gateway
AWS Storage Gateway | Storage
Your NFS client can access file system metadata including ownership, permissions, and timestamps, for files and directories. Ownership, permissions, and timestamps, are durably stored in S3 in the user-metadata of the object associated with the file. You can configure default ownership and permissions for objects which were stored in the S3 bucket directly, not by file gateway.
How do I set the Content-Type for files uploaded to S3?
File Gateway
AWS Storage Gateway | Storage
For each file share, you can enable guessing of MIME types for uploaded objects upon creation or enable the feature later. If enabled, file gateway will use the filename extension to determine the MIME type for the file and set the S3 objects Content-Type accordingly. This is beneficial if you are using file gateway to manage objects in S3 which you access directly via URL or distribute through Amazon CloudFront.
Can I use multiple NFS clients with a single file gateway?
File Gateway
AWS Storage Gateway | Storage
You can have multiple NFS clients accessing a single file gateway. However, as with any NFS server, concurrent modification from multiple NFS clients can lead to unpredictable behavior. Application level coordination is required to do this in a safe way.
Can I have multiple writers to my S3 bucket?
File Gateway
AWS Storage Gateway | Storage
No. We recommend a single writer to objects in your S3 bucket. If you directly overwrite or update an object previously written by file gateway, it results in undefined behavior when the object is accessed through the file share. Concurrent modification of the same object (e.g. via the S3 API and the file gateway) can lead to unpredictable results and we recommend against this configuration.
Can I have two gateways writing independent data to the same bucket?
File Gateway
AWS Storage Gateway | Storage
We do not recommend configuring multiple writers to a single bucket because it can lead to unpredictable results. You could enforce unique object names or prefixes through your application workflow. File gateway doesn’t monitor or report on conflicts in such a setup.
Can I have multiple gateways reading data from the same bucket?
File Gateway
AWS Storage Gateway | Storage
Yes, you can have multiple readers on a bucket managed through a file gateway. You can configure a file share as read-only, and allow multiple gateways to read objects from the same bucket. Additionally, you can refresh the inventory of objects that your gateway knows about using the RefreshCache API.
Note however that the if you do not configure a file share as read-only, file gateway does not monitor or restrict these readers from inadvertently writing to the bucket. It is up to you to maintain a single writer/multi reader configuration from your application.
How do I know when my file is uploaded?
File Gateway
AWS Storage Gateway | Storage
When you write files to your file share, the data is stored locally first and then asynchronously uploaded to your S3 bucket. You can request notification through AWS CloudWatch Events when this upload completes. These notifications can be used to trigger additional workflows, such as invoking an AWS Lambda function or Amazon EC2 Systems Manager Automation, which is dependent upon the data that is now available in S3. To learn more, please refer to the documentation.
What if my bucket already contains objects?
File Gateway
AWS Storage Gateway | Storage
If your bucket already contains objects when you configure it for use with file gateway, they will appear as files to NFS clients. The files are given default file system metadata. These defaults are configurable.
To reduce latency and S3 requests, file gateway only scans the object headers when you explicitly list the files or directories. Data is downloaded only when the entire object is read.
Can I detect new objects created in my S3 bucket after the file share has been created (e.g. created from an object-based workload)?
File Gateway
AWS Storage Gateway | Storage
You can refresh the inventory of objects that your gateway knows about using the RefreshCache API and console action.
Can I directly access objects stored in S3 by using file gateway?
File Gateway
AWS Storage Gateway | Storage
Yes. Once objects are stored in S3, you can access them directly in AWS for in-cloud workloads without requiring file gateway. Your objects inherit the properties of the S3 bucket in which they are stored, such as lifecycle management, and cross-region replication.
An object that needs to be accessed by using a file share should only be managed by the gateway. If you directly overwrite or update an object previously written by file gateway, it results in undefined behavior when the object is accessed through the file share.
How many files shares can I create per bucket?
File Gateway
AWS Storage Gateway | Storage
There is a one-to-one mapping between a file share and a bucket. We do not limit the number of file shares per bucket. However, we recommend having a single writer to the bucket, either a file gateway or client accessing S3 directly.
How many file shares can I create per gateway?
File Gateway
AWS Storage Gateway | Storage
You can create up to 10 file shares per gateway.
What is the maximum size of an individual file?
File Gateway
AWS Storage Gateway | Storage
The maximum size of an individual file is 5 TB, which is the maximum size of an individual object in S3. If you write a file larger than 5 TB, you will get a “file too large” error message and only the first 5 TB of the file will be uploaded.
My application checks storage size before copying data. What storage size does the gateway return?
File Gateway
AWS Storage Gateway | Storage
The gateway returns a large number (8 EB) as your total capacity. Amazon S3 does not limit total storage.
Can I use versioning, lifecycle, cross-region replication, and S3 event notification?
File Gateway
AWS Storage Gateway | Storage
Yes. Your bucket policies for versioning, lifecycle management, cross-region replication, and S3 event notification, apply directly to objects stored in your bucket through AWS Storage Gateway.
You can use S3 lifecycle policies to change an object’s storage tier or delete old objects or object versions. In the case of objects deleted by lifecycle policy, you will need to call the RefreshCache API to reflect these changes to your NFS clients.
When using an S3 bucket which is the target for cross-region replication, you may need to use the RefreshCache API to ensure the gateway cache and S3 bucket are in sync.
If using S3 event notifications you may receive events for partial files created by the gateway to ensure your data is durably stored in S3. Partial files may occur for a number of reasons, such as the gateway needing to free up cache space, or a high rate of writes to a file. These partial files may not be application consistent.
Can I read and write files directly to Amazon Glacier?
File Gateway
AWS Storage Gateway | Storage
Your bucket lifecycle policies will enable you to move files to Amazon Glacier. If you read a file that is in Amazon Glacier, you will receive a generic I/O error.
