Plato is a Linux-based, heterogeneous, high-performance computing (HPC) cluster at the University of Saskatchewan. It is used for research and for training, and is restricted to USask users and their collaborators. Plato is managed by ICT’s Advanced Research Computing (ARC) team.

Although it is not a Digital Research Alliance of Canada cluster, Plato is configured to be as similar as possible to Alliance clusters. For instance, Plato and Alliance clusters use the same software stack and job scheduler.

Plato cannot provide all the computing power required for USask research projects. Researchers with important computing needs should consider Plato a stepping stone to test their projects locally before moving to Alliance clusters.

Basic information

Storage

Network

Most Plato compute nodes are interconnected by a 1Gb Ethernet link. Login nodes are connected to compute nodes and to the University network by a 10Gb Ethernet link. Some compute nodes (see below) are interconnected by 10Gb Ethernet or FDR InfiniBand. Both login and compute nodes can access the external network, including the Internet.

Scheduler

Plato uses the SLURM scheduler. Job duration is limited to 21 days, except for the GPU and large-memory nodes where the limit is 7 days. (This limit does not apply to contributed hardware.) Shorter jobs get increased priority, according to the categories below. The default allocation is 20 minutes for a single task on one CPU with 512M of memory. Job arrays are limited to 10000 steps. Jobs can only be submitted from login nodes, not from compute nodes; it is therefore not possible to submit a job from within another.

Architecture

The oldest CPU model in use on Plato is from the Intel Ivy Bridge Xeon product line. These processors support the AVX instruction set, but not any later instruction sets (such as AVX2 or AVX512). When compiling your code on a login node, use GCC with the -march=core-avx option to get the best performance while retaining compatibility with all Plato compute nodes. Do not use -march=native as this would produce code that would be incompatible with Ivy Bridge processors.

Plato uses the AVX branch of the Compute Canada software stack. Therefore, the available software modules can differ slightly from those available on Compute Canada clusters, that typically use the AVX2 branch of the software stack. We can add missing software and versions on request.

Compute nodes

Public	Count	Type	CPU	Cores / Node	Highest SIMD	Usable Memory / Node	GPU	/local storage	Interconnect
	64	“Pipit”	2 x Intel Xeon E5-2640 v2 @ 2.00GHz “Ivy Bridge”	16	AVX	31000M (30G)		347G	1Gb Ethernet
	28	“Penguin”	2 x Intel Xeon Gold 6148 @ 2.40GHz “Skylake”	40	AVX512	190000M (185G)		781G	10Gb Ethernet
	4	Large-memory “Penguin”	2 x Intel Xeon Gold 6148 @ 2.40GHz “Skylake”	40	AVX512	384000M (375G)		781G	10Gb Ethernet
	2	GPU	2 x Intel Xeon E5-2640 v3 @ 2.60GHz “Haswell”	16	AVX2	31000M (30G)	2 x NVIDIA K40	805G	1Gb Ethernet
	20	GWF	2 x Intel Xeon E5-2640 v2 @ 2.00GHz “Ivy Bridge”	16	AVX	31000M (30G)		347G	1Gb Ethernet
	2	Tse group	2 x Intel Xeon E5-2683 v4 @ 2.10GHz “Broadwell”	32	AVX2	250000M (244G)	2 x NVIDIA K80	768G	FDR InfiniBand

Choosing nodes

Plato will choose the appropriate node type for your job according to your resource requirements (cores per node, memory, GPUs). It is therefore not necessary to request a specific node type. Doing so may reduce the number of nodes eligible to run your job, increasing your wait time. You can specify a node type using SLURM options: --constraint=ivybridge (for Pipit nodes) or --constraint=skylake (for Penguin nodes).

Login nodes

Plato has two login nodes, accessible by SSH at plato.usask.ca. The login nodes should be used to prepare jobs and submit them to the scheduler, to compile programs, and to run short calculations that require little memory and processing power. Intensive processes must never be run on the login node; they must be submitted to the scheduler to run on the compute nodes.

File transfers

Files can be transferred between Plato and your local computer using the SCP protocol through standard commands such as scp, rsync, etc. There are also Globus endpoints for home directories (USASK-GLOBALHOME) and Datastore (USASK-DATASTORE). Always use Globus for large data transfers between /globalhome and /datastore.

Educational accounts

If you are granted access to Plato for a class or training workshop, your jobs will be limited to 12 hours of runtime, and you will not have access to the large-memory nodes.