Parallelcluster Flashcards
Explain what it is and the value prop
Open source cluster management tool for HPC on AWS. Cluster defined as a system with a head node, compute nodes, a job scheduler, and a shared file system. Makes managing clusters easy, automates resource scaling
What operating systems does it support
Amazon Linux 2, Ubuntu 18.04, CentOS 7, RHEL8, and CentOS 8
What schedulers does it support
Slurm and Batch. No support for SGE or Torque
How does Ansible differ from PCluster
Ansible will provision software on the cluster, but it doesn’t provision the head node or compute nodes themselves. Like actually launching the instances. But to install everything in the Pcluster env, like EFA, Slurm, etc it uses Chef. So if you have Ansible to provision your cluster, you can actually run this on Pcluster just by removing the portions that control Slurm and some of the other core functionality that pcluster already does.
Will an EC2 OS update interrupt my workloads
No, ParallelCluster disables cluster instance automatic OS updates. You can do this manually if you want.
Can I start and stop a head node
Yes, as long as the instance type doesnt have an instance store
my customer has built out parallelclusterUI successfully in a VPC with no internet egress (very strict rules in their organisation), and they’re now testing with some simple benchmarking scripts/tools however as soon as they attempt to run compute nodes they just hang with status “Pending” and then terminate and it tries again. Is there a way to troubleshoot what is happening and why its not able to instantiate compute nodes successfully?
This workshop content walks through cluster configuration (DisableManagedDns and UseEc2Hostnames),
You also need VPC Endpoints for various services see the table on this page or if you need an example CFN template for how to setup the VPC endpoints, review this
If you have external storage (i.e., storage not managed by ParallelCluster) linked on the cluster SharedStorage config, you’ll also need to make sure you have the right security groups on AdditionalSecurityGroups for both head and compute nodes.
Assuming all of your configuration is setup properly, review /var/log/parallelcluster/slurm_resume.log on the headnode to confirm whether there is an Insufficient Capacity error preventing compute node spin-up.
Can my cluster be in a private subnet? What about launching the cluster?
If the cluster is operational and launched, it can be in a completely private subnet. However, when you install the PCluster tool you still have to go out and get the python package which would be over the Internet. This CAN be pre positioned inside an S3 bucket which contains the binary installer, however, think of other regions.
How many queues can I create
50
What scheduler supports multiple queues
Slurm only
How are hostnames assigned to the compute nodes I launch
Both hostnames and fully-qualified domain names (FQDN) are created using Amazon Route 53 hosted zones.
From the example configuration file above, a given node from queue1 and compute resource c5xlarge has a hostname: queue1-dy-c5xlarge-1.
What is the purpose of the DDB table
Used internally in the cluster to keep a mapping between node names and instance ids. Basically every time a node is launched, its mapping is registered in that table.
How can I monitor my cluster
Using CloudWatch you can get metrics such as mem_used_percent and disk_used_percent. We also track common errors of compute nodes on the CW dashboard.
How does Parallelcluster scale
It does NOT use auto scaling anymore. We are now calling EC2 APIs to run and terminate instances based on the scheduler workload. Moreover we have centralized all the cluster management operations on the head node so that we can batch as many nodes as possible in a single API call.
Slurm power management plugin is crucial here. Before, when you submitted a job the plugin would basically say I need more nodes. It would increment the auto scaling group desired count. With auto scaling you can set min, max, desired. So auto scaling was making the API call to EC2 to spin up more instances. This would ensure a failed node would be replaced automatically.
Today, the PMP calls the run instances API directly. The reason we do this is that it allows us to launch a fleet of nodes at the same time. With auto scaling it would launch 4,8,etc and then go to sleep for a bit (idle timeout), and then go through another iteration. The PMP also can check status of nodes (health check).
Why would EBS volumes be used for shared storage
Are attached to the head node and shared with compute nodes through NFS. This option can be cost effective, but performance depends on the head node resources as storage needs scale. This can become a bottleneck as more compute nodes are added to the cluster and the throughput demand increases.
A lot of CFD work can get back doing a shared EBS volume on the head node. It might run slow depending on IO pattern but it will work. With CFD you have a little bit more time compared to something like weather forecasting. Lustre would be important for this to write out results.
