On Premise Tools Flashcards
What is GPFS
GPFS and Lustre (FSx for Lustre is Amazon’s managed Lustre service) are both parallel file systems designed for high-performance computing environments, and they share many similarities in terms of their features and capabilities. However, there are also some key differences between the two systems.
One major difference is that GPFS is a proprietary file system developed by IBM, while Lustre is an open-source project originally developed by Sun Microsystems and now maintained by a community of developers. As an open-source project, Lustre is more flexible in terms of customization and can be modified and adapted to meet specific user needs.
Another difference is that GPFS is designed to be more scalable than Lustre. GPFS can handle extremely large file systems and can support tens of thousands of clients accessing the file system simultaneously. Lustre, on the other hand, is typically used in smaller clusters and can support up to a few thousand clients.
In terms of performance, both GPFS and Lustre are highly parallel and can provide very fast access to data. However, the specific performance characteristics of each system can depend on the specific hardware and software configurations being used.
Finally, GPFS includes a number of advanced features for managing and monitoring file system performance, such as performance monitoring and tuning tools, and support for various access protocols. FSx for Lustre also includes similar features, but these are offered as a managed service by Amazon.
Overall, both GPFS and Lustre are highly capable parallel file systems designed for high-performance computing environments. The choice between the two will depend on the specific needs of the user, including factors such as scalability requirements, customization needs, and available resources.
How can I sync data on prem with the cloud
Yes, it is possible to synchronize data between an on-premises GPFS file system and FSx for Lustre in AWS using various methods, such as AWS DataSync or AWS Storage Gateway.
AWS DataSync is a managed data transfer service that can be used to move large amounts of data between on-premises storage systems and AWS storage services. It supports a variety of sources and targets, including NFS, SMB, and S3, as well as Lustre file systems. With DataSync, you can schedule and automate data transfers, monitor transfer progress, and verify the integrity of the transferred data.
AWS Storage Gateway is another service that can be used to bridge on-premises storage with AWS storage services. It provides a virtual machine that can be installed on-premises and acts as a gateway to AWS storage services. It supports various protocols, including NFS and SMB, and can be used to cache frequently accessed data locally while storing less frequently accessed data in AWS storage services.
To synchronize data between GPFS and FSx for Lustre, you could use either of these services to transfer data from your on-premises GPFS file system to an FSx for Lustre file system in AWS. Once the initial transfer is complete, you could use either service to keep the two file systems in sync by transferring changed or new data on a regular basis. The specific method used would depend on your specific needs and requirements, as well as the size and frequency of data transfers.
What are other ways I can sync data between on premise and AWS
To use FileCache in a hybrid environment, you can use AWS Direct Connect or VPN to create a secure, high-speed connection between your on-premises data center and your AWS VPC. Once you have established the connection, you can deploy FileCache in your VPC to cache frequently accessed files from your on-premises storage.
When your application running on your on-premises infrastructure needs to access a file that is also stored in the cache, it can fetch the file from the cache instead of fetching it from the on-premises storage, which can significantly reduce latency and improve performance.
However, keep in mind that deploying and configuring FileCache in a hybrid architecture requires careful planning and setup. You will need to ensure that your network infrastructure is properly configured and secured to allow traffic to flow between your on-premises environment and AWS. You will also need to ensure that your cache is properly sized and configured to meet your performance requirements.
Overall, using Amazon FileCache in a hybrid architecture can be an effective way to improve the performance of applications that span both on-premises and cloud-based infrastructure.
Is file cache POSIX
Yes
How many file systems can I link with File Cache
You can link up to eight NFS file systems or eight S3 buckets to a cache, and they will be exposed as a unified set of files and directories
What kind of performance does this get me
When using File Cache, your applications benefit from consistent, sub-millisecond latencies, up to hundreds of GB/s of throughput, and up to millions of operations per second. Just like with other storage services, such as Amazon Elastic Block Store (Amazon EBS), the performance depends on the size of the cache. The cache size can be expanded to petabyte scale, with a minimum size of 1.2 TiB.
Does Slurm have fair share scheduling
Yes and it is enabled by default. There is a way to do a multi factor fair share scheduling where you can weigh jobs based on different competing interests. Things like number of jobs that have run already vs jobs that are in the queue vs how much capacity you have in the cluster. This multi priority plugin is something you have to enable yourself.
How can I edit scheduling settings for Slurm
Slurm config level
What factors are at play for fair share
size of queue and the sources that could execute on the compute fleet, and execute based on job priority and other factors
