Magic Castle is an open-source software that aims to recreate the HPC user experience in public clouds. It uses the open-source software Terraform and HashiCorp Language (HCL) to define the virtual machines, volumes, and networks that are required to replicate a virtual HPC infrastructure. In this context, an infrastructure definition refers to a collection of Terraform configuration files that describe the desired state of the cluster - including the number and types of virtual machines, networking topology, storage configuration, and access control policies - essentially a blueprint for the entire cluster.
The infrastructure definition is packaged as a Terraform module that users can customize as they require. After deployment, the user is provided with a complete HPC cluster software environment including a Slurm scheduler, a Globus Endpoint, JupyterHub, LDAP, DNS, and over 3000 research software applications compiled by experts with EasyBuild. Magic Castle is compatible with AWS, Microsoft Azure, Google Cloud, OpenStack, and OVH.
Magic Castle can also be deployed in containers using Incus.
We believe in the following precepts about HPC:
- Everyone is entitled to learn HPC.
- To learn HPC, you have to practice HPC.
- To practice HPC, you need access to HPC infrastructure.
- HPC infrastructure can be configured in cloud, community or commercial.
- Moving HPC infrastructure between cloud providers should be painless.
We built Magic Castle so neophytes can create their own sandbox to learn and experiment with HPC without asking permission to. And once they know enough, they can scale up their environment and address real-world issues, anywhere.