AWS Storage Gateway | File Gateway

Question 1

Is AWS Storage Gateway HIPAA eligible?

File Gateway

AWS Storage Gateway | Storage

Answer 1

Yes. AWS Storage Gateway is HIPAA eligible. If you have an executed Business Associate Agreement (BAA) with AWS, you can use Storage Gateway to store, backup and archive protected health information (PHI) on scalable, cost-effective, and secure AWS storage services, including Amazon S3, Amazon Glacier and Amazon EBS, which are also HIPAA eligible.

Information on HIPAA eligible services on AWS can be found at our HIPAA Compliance page, and you can enter into a BAA with AWS here. HIPAA eligibility for Storage Gateway applies to all gateway types (File, Volume and Tape).

Question 2

What is file gateway?

File Gateway

AWS Storage Gateway | Storage

Answer 2

File gateway provides a virtual on-premises file server, which enables you to store and retrieve Amazon S3 objects through standard file storage protocols.

Question 3

What can I do with file gateway?

File Gateway

AWS Storage Gateway | Storage

Answer 3

The use cases for file gateway include: (a) ingesting file-based data into S3 for object workloads, (b) “cloud-bursting”, where you can move data to AWS, run a workload, and access results within your on-premises application, (c) migrating file to objects in S3, (d) cost-effective storage for backup and archive, (f) disaster recovery, and (g) tiering on-premises file storage to S3.

Question 4

What are the benefits of using file gateway to store data in S3?

File Gateway

AWS Storage Gateway | Storage

Answer 4

File gateway enables your existing file-based applications, devices, and workflows to use cloud storage without modification. File gateway securely and durably stores both file contents and metadata as objects in your Amazon S3 buckets using standard file protocols.

Question 5

What protocols are supported? Can I use Microsoft Windows?

File Gateway

AWS Storage Gateway | Storage

Answer 5

File gateway supports clients connecting to the gateway using NFS v3 and v4.1. Microsoft Windows clients that support NFS v3 can connect to file gateway.

Question 6

How do I create and configure a file share?

File Gateway

AWS Storage Gateway | Storage

Answer 6

You can create a file share using the AWS Management Console or service API, and associate it with a new or existing S3 bucket. You can configure your file share with administative controls such as limiting acces to specific NFS clients or networks, read-only or read-write, or enabling user permisssion squashing.

Question 7

How does file gateway access my S3 bucket?

File Gateway

AWS Storage Gateway | Storage

Answer 7

File gateway uses an AWS Identity and Access Management (IAM) role to access your S3 bucket. You can set this up yourself, or have it automatically setup by the AWS Storage Gateway Management Console. For automatic setup, AWS Storage Gateway will create a new IAM role in your account and associate it with an IAM Access Policy to access your S3 bucket. The IAM role and IAM access policy are created in your account and you can fully manage them yourself.

Question 8

How does my application access my file share?

File Gateway

AWS Storage Gateway | Storage

Answer 8

To use the file share, you mount it from your application using standard UNIX or Windows commands. For convenience, example command lines are shown in the management console

Question 9

What is the relationship between files and objects?

File Gateway

AWS Storage Gateway | Storage

Answer 9

Files are stored as objects in your S3 buckets and you can configure the initial storage class for objects that file gateway creates. There is a one-to-one relationship between files and objects, and you can configure the initial storage class for objects that file gateway creates.

The object key is derived from the file path within the file system. For example, if you have a gateway with hostname file.amazon.com and have mapped my-bucket, then file gateway will expose a mount point called file.amazon.com:/export/my-bucket. If you then mount this locally on /mnt/my-bucket and create a file named file.html in a directory /mnt/my-bucket/dir this file will be stored as an object in the bucket my-bucket with a key of dir/file.html.

Question 10

What file system operations are supported by file gateway?

File Gateway

AWS Storage Gateway | Storage

Answer 10

Your NFS client can create, read, update, and delete, files and directories. Files are stored as individual objects in S3. Directories are managed as folder objects in S3, using the same syntax as the S3 console.

Symbolic links and hard links are not supported. Attempting to create a link will result in an NFS error. Creating sparse files will result in a non-sparse zero-filled object in S3.

Rename operations will appear atomic to NFS clients, but S3 does not support renaming of objects. When you rename a file or directory the gateway performs copy-put requests to create a copy of the objects in S3 under the new keys and then deletes the originals. This avoids having to re-send large files over the network. Renaming directories containing a large number of files is not instantaneous and will result in 2 copies of your data being stored in S3 until the rename operation completes.

Question 11

What file system metadata can my NFS client access?

File Gateway

AWS Storage Gateway | Storage

Answer 11

Your NFS client can access file system metadata including ownership, permissions, and timestamps, for files and directories. Ownership, permissions, and timestamps, are durably stored in S3 in the user-metadata of the object associated with the file. You can configure default ownership and permissions for objects which were stored in the S3 bucket directly, not by file gateway.

Question 12

How do I set the Content-Type for files uploaded to S3?

File Gateway

AWS Storage Gateway | Storage

Answer 12

For each file share, you can enable guessing of MIME types for uploaded objects upon creation or enable the feature later. If enabled, file gateway will use the filename extension to determine the MIME type for the file and set the S3 objects Content-Type accordingly. This is beneficial if you are using file gateway to manage objects in S3 which you access directly via URL or distribute through Amazon CloudFront.

Question 13

Can I use multiple NFS clients with a single file gateway?

File Gateway

AWS Storage Gateway | Storage

Answer 13

You can have multiple NFS clients accessing a single file gateway. However, as with any NFS server, concurrent modification from multiple NFS clients can lead to unpredictable behavior. Application level coordination is required to do this in a safe way.

Question 14

Can I have multiple writers to my S3 bucket?

File Gateway

AWS Storage Gateway | Storage

Answer 14

No. We recommend a single writer to objects in your S3 bucket. If you directly overwrite or update an object previously written by file gateway, it results in undefined behavior when the object is accessed through the file share. Concurrent modification of the same object (e.g. via the S3 API and the file gateway) can lead to unpredictable results and we recommend against this configuration.

Question 15

Can I have two gateways writing independent data to the same bucket?

File Gateway

AWS Storage Gateway | Storage

Answer 15

We do not recommend configuring multiple writers to a single bucket because it can lead to unpredictable results. You could enforce unique object names or prefixes through your application workflow. File gateway doesn’t monitor or report on conflicts in such a setup.

Question 16

Can I have multiple gateways reading data from the same bucket?

File Gateway

AWS Storage Gateway | Storage

Answer 16

Yes, you can have multiple readers on a bucket managed through a file gateway. You can configure a file share as read-only, and allow multiple gateways to read objects from the same bucket. Additionally, you can refresh the inventory of objects that your gateway knows about using the RefreshCache API.

Note however that the if you do not configure a file share as read-only, file gateway does not monitor or restrict these readers from inadvertently writing to the bucket. It is up to you to maintain a single writer/multi reader configuration from your application.

Question 17

How do I know when my file is uploaded?

File Gateway

AWS Storage Gateway | Storage

Answer 17

When you write files to your file share, the data is stored locally first and then asynchronously uploaded to your S3 bucket. You can request notification through AWS CloudWatch Events when this upload completes. These notifications can be used to trigger additional workflows, such as invoking an AWS Lambda function or Amazon EC2 Systems Manager Automation, which is dependent upon the data that is now available in S3. To learn more, please refer to the documentation.

Question 18

What if my bucket already contains objects?

File Gateway

AWS Storage Gateway | Storage

Answer 18

If your bucket already contains objects when you configure it for use with file gateway, they will appear as files to NFS clients. The files are given default file system metadata. These defaults are configurable.

To reduce latency and S3 requests, file gateway only scans the object headers when you explicitly list the files or directories. Data is downloaded only when the entire object is read.

Question 19

Can I detect new objects created in my S3 bucket after the file share has been created (e.g. created from an object-based workload)?

File Gateway

AWS Storage Gateway | Storage

Answer 19

You can refresh the inventory of objects that your gateway knows about using the RefreshCache API and console action.

Question 20

Can I directly access objects stored in S3 by using file gateway?

File Gateway

AWS Storage Gateway | Storage

Answer 20

Yes. Once objects are stored in S3, you can access them directly in AWS for in-cloud workloads without requiring file gateway. Your objects inherit the properties of the S3 bucket in which they are stored, such as lifecycle management, and cross-region replication.

An object that needs to be accessed by using a file share should only be managed by the gateway. If you directly overwrite or update an object previously written by file gateway, it results in undefined behavior when the object is accessed through the file share.

Question 21

How many files shares can I create per bucket?

File Gateway

AWS Storage Gateway | Storage

Answer 21

There is a one-to-one mapping between a file share and a bucket. We do not limit the number of file shares per bucket. However, we recommend having a single writer to the bucket, either a file gateway or client accessing S3 directly.

Question 22

How many file shares can I create per gateway?

File Gateway

AWS Storage Gateway | Storage

Answer 22

You can create up to 10 file shares per gateway.

Question 23

What is the maximum size of an individual file?

File Gateway

AWS Storage Gateway | Storage

Answer 23

The maximum size of an individual file is 5 TB, which is the maximum size of an individual object in S3. If you write a file larger than 5 TB, you will get a “file too large” error message and only the first 5 TB of the file will be uploaded.

Question 24

My application checks storage size before copying data. What storage size does the gateway return?

File Gateway

AWS Storage Gateway | Storage

Answer 24

The gateway returns a large number (8 EB) as your total capacity. Amazon S3 does not limit total storage.

Question 25

Can I use versioning, lifecycle, cross-region replication, and S3 event notification?

File Gateway

AWS Storage Gateway | Storage

Answer 25

Yes. Your bucket policies for versioning, lifecycle management, cross-region replication, and S3 event notification, apply directly to objects stored in your bucket through AWS Storage Gateway.

You can use S3 lifecycle policies to change an object’s storage tier or delete old objects or object versions. In the case of objects deleted by lifecycle policy, you will need to call the RefreshCache API to reflect these changes to your NFS clients.

When using an S3 bucket which is the target for cross-region replication, you may need to use the RefreshCache API to ensure the gateway cache and S3 bucket are in sync.

If using S3 event notifications you may receive events for partial files created by the gateway to ensure your data is durably stored in S3. Partial files may occur for a number of reasons, such as the gateway needing to free up cache space, or a high rate of writes to a file. These partial files may not be application consistent.

Question 26

Can I read and write files directly to Amazon Glacier?

File Gateway

AWS Storage Gateway | Storage

Answer 26

Your bucket lifecycle policies will enable you to move files to Amazon Glacier. If you read a file that is in Amazon Glacier, you will receive a generic I/O error.

Question 27

Can I use file gateway with my backup application?

File Gateway

AWS Storage Gateway | Storage

Answer 27

File gateway supports NFS v3 and v4.1 clients. We are continuing to do on-going testing with common backup apps. Please provide us specific apps you’d like to see compatibility with.

Question 28

Can I use file gateway to write files to EFS?

File Gateway

AWS Storage Gateway | Storage

Answer 28

No. File gateway allows you to write files as objects in S3.

Question 29

When should I use file gateway vs. the S3 API?

File Gateway

AWS Storage Gateway | Storage

Answer 29

You can use file gateway when you want to access objects in S3 as files using standard filesystem operations. File gateway additionally provides low-latency local access and efficient data transfer. You can use the S3 API when your application doesn’t require file system operations and can manage data transfer directly.

Question 30

How does file gateway manage the local cache? What data gets stored locally?

File Gateway

AWS Storage Gateway | Storage

Answer 30

Local disk storage on the gateway is used to temporarily hold changed data that needs to be transferred to AWS, and to locally cache data for low-latency read access. File gateway automatically manages the cache maintaining the most recently accessed data based on NFS client read and write operations. Data is evicted from the cache only when space is needed to store more recently used data.

To maximize write performance, the gateway uses a write-back mechanism where data is first persisted to disk and then asynchronously uploaded to S3. The gateway serves data from the local cache to maximize read performance. If not present, data is efficiently synchronously fetched from Amazon S3 using byte-range gets.

The local cache should generally be sized for the working set of data that you need low-latency access to. If the cache is too small then read latencies will increase as data being requested must be fetched from S3, and writes could fail if there is no free cache space to store data locally pending upload to S3.

Question 31

Does file gateway perform data reduction (deduplication or compression)?

File Gateway

AWS Storage Gateway | Storage

Answer 31

No. Files are mapped to objects one-to-one in your bucket without modification, enabling you to access your data directly in S3 without needing to use the gateway or deploy additional software to rehydrate your data.

File gateway uses multipart uploads and copy put, so only changed data is uploaded to S3 which can reduce data transfer. The gateway does not automatically download fulle objects or all the data that exists in your bucket; data is only downloaded when explicitly accessed by your NFS client.

Question 32

Can I use file gateway with Amazon S3 Transfer Acceleration?

File Gateway

AWS Storage Gateway | Storage

Answer 32

If your bucket is configured for S3 Transfer Acceleration, file gateway will not use the accelerated endpoints.

Question 33

How does file gateway read and write my S3 bucket?

File Gateway

AWS Storage Gateway | Storage

Answer 33

File gateway uses an IAM role to access your S3 bucket. You can create this role yourself, or use the automated mechanism provided in the AWS Management Console. For automatic role creations, file gateway will create a new IAM role in your account and associate this role with an IAM access policy to access your S3 bucket. The IAM role and IAM access policy are created in your account, and you can fully manage them.