If you want to do the same thing, build a cluster of complete machines, in a ligh-weight manner, just for the learning experience, you can use incus to create containers in the same manner.
As they are all complete machines and you bridge them, you get DHCP allocation identical to running a little PC on the local LAN. If you have ssh set up in the image, as described, you can run the identical scripts.
As a plus, if you run them on ZFS with de-dup, even the disk cost for new machines is miniscule.
I appreciate the author's work in doing this and writing it all up so nicely. However every time I see someone doing this, I cannot help but wonder why they are not just using SLURM + Nextflow. SLURM can easily cluster the separate computers as worker nodes, and Nextflow can orchestrate the submission of batch jobs to SLURM in a managed pipeline of tasks. The individual tasks to submit to SLURM would be the users's own R scripts (or any script they have). Combine this with Docker containers to execute on the nodes to manage dependencies needed for task execution. And possibly Ansible for the management of the nodes themselves to install the SLURM daemons and packages etc.. Taken together this creates a FAR more portable and system-agnostic and language-agnostic data analysis workflow that can seamlessly scale over as many nodes and data sets as you can shove into it. This is a LOT better than trying to write all this code in R itself that will do the communication and data passing between nodes directly. Its not clear to me that the author actually needs anything like that, and whats worse, I have seen other authors write exactly that in R and end up re-inventing the wheel of implementing parallel compute tasks (in R). Its really not that complicated. 1) write R script that takes a chunk of your data as input, processes it, writes output to some file, 2) use a workflow manager to pass in chunks up the data to discrete parallel task instances of your script / program and submit the tasks as jobs to 3) a hardware-agnostic job scheduler running on your local hardware and/or cloud resources. This is basically the backbone of HPC but it seems like a lot of people "forget" about the 'job scheduler' and 'workflow manager' parts and jump straight to glueing data-analysis code to hardware. Also important to note that most all robust workflow managers such as Nextflow also already include the parts such as "report task completion", "collect task success / failure logs", "report task CPU / memory resource usages", etc.. So that you, the end user, only need to write the parts that implement your data analysis.
Something like proxmox [1] could make wiping everything and restarting a lot easier. There really isn't a huge penalty between bare metal and a VM now so you get the ability to create/deploy/monitor all from a reasonable interface. If the standard clustering stuff isn't enough their datacenter version looks like it is designed more for this. Never used it though. (no ties to proxmox here, just a guy that likes it!)
I'm planning to set up something similar (but for compiling code). The difference is that my systems came without storage, so I intend to netboot them, which adds a whole other level of complication. I'm planning to use NixOS, with something like nixos-netboot-serve[0].
Congratulations on learning about distributed electronic computers. (This is worth tinkering with. This is how people actually get good at HPC.)
Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings.
[Snark]
In practice:
With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake:
Something I've been playing with: what's the cheapest semi-reasonable computer that I could make a toy cluster from? Currently eyeing buying a bunch of Raspberry Pi Zeros but I suspect there are cheaper options (maybe some tiny openwrt thing). Too bad ESP32s don't have an MMU:D
Really depends on the details of your goals, but why not just VMs?
The "problem" with Pi-like devices is that they're usually not very "normal". The process of provisioning is different, IIRC they only "recently" supported booting off something other than the SD Card, and in the case of the Zeros, you'll either be using Wifi or an external USB Ethernet dongle (over USB OTG no less). Sometimes they need specially compiled version of linux, so you're stuck far from mainline (this was a big component of the RPis success) This may be distracting from your goals of learning about clustering.
I suspect the $10 Pi Zero is about as cheap as you'll get though, depending on your personal costs of case + ethernet dongle + USB power supply, etc.
I have a bunch of old Rpi 3B's in a 19" 1RU rack space i use for toy clustering. PoE splitters from a PoE switch to power them which also provides the network uplink. All very neat and self contained and when it comes times to upgrade/replace if they all fail (i have a couple of spares) I can just pull out the 1RU for disposal.
Intel NUC's are probably much better value for money these days but the 3B's were pretty cheap at the time I bought them.
You can get intel NUCs way cheaper if you look around, or the Lenovo mini PCs. Small clusters will never beat a decent CPU but you can probably make a cluster of old mini PCs for less than the price of one.
So obviously it depends on goals; I absolutely agree that up to a surprisingly high point, the best way to actually get the most compute is to go buy the most powerful single machine that will do what you want off eBay or whatever. My goal is the opposite; I want to specifically get my hands dirty figuring out how to actually provision and manage and operate a large number of hosts, even if they're so low resource as to be effectively worthless individually and are still weak clustered. To that end specifically, digikey claims to be willing to sell me a rather large number of pi zero 2w at $15/each - https://www.digikey.com/en/products/detail/raspberry-pi/SC11... - and even cheap used boxes on eBay seem to start at double that. Obviously you need a little bit more to actually run them, but I believe the only thing you actually need is a USB cable and then a single host computer can provide power and boot them over usb.
Just be aware that you’ll be stuck with wifi, or spending far more on ethernet adapters for them. If you want cheap with ethernet there’s other devices supported by Arabian in the same price range which have ethernet. It’s fun for sure to have a huge number of very cheap machines, I have an old arm cluster made of Odroid boards for that purpose.
If you want to do the same thing, build a cluster of complete machines, in a ligh-weight manner, just for the learning experience, you can use incus to create containers in the same manner. As they are all complete machines and you bridge them, you get DHCP allocation identical to running a little PC on the local LAN. If you have ssh set up in the image, as described, you can run the identical scripts.
As a plus, if you run them on ZFS with de-dup, even the disk cost for new machines is miniscule.
I appreciate the author's work in doing this and writing it all up so nicely. However every time I see someone doing this, I cannot help but wonder why they are not just using SLURM + Nextflow. SLURM can easily cluster the separate computers as worker nodes, and Nextflow can orchestrate the submission of batch jobs to SLURM in a managed pipeline of tasks. The individual tasks to submit to SLURM would be the users's own R scripts (or any script they have). Combine this with Docker containers to execute on the nodes to manage dependencies needed for task execution. And possibly Ansible for the management of the nodes themselves to install the SLURM daemons and packages etc.. Taken together this creates a FAR more portable and system-agnostic and language-agnostic data analysis workflow that can seamlessly scale over as many nodes and data sets as you can shove into it. This is a LOT better than trying to write all this code in R itself that will do the communication and data passing between nodes directly. Its not clear to me that the author actually needs anything like that, and whats worse, I have seen other authors write exactly that in R and end up re-inventing the wheel of implementing parallel compute tasks (in R). Its really not that complicated. 1) write R script that takes a chunk of your data as input, processes it, writes output to some file, 2) use a workflow manager to pass in chunks up the data to discrete parallel task instances of your script / program and submit the tasks as jobs to 3) a hardware-agnostic job scheduler running on your local hardware and/or cloud resources. This is basically the backbone of HPC but it seems like a lot of people "forget" about the 'job scheduler' and 'workflow manager' parts and jump straight to glueing data-analysis code to hardware. Also important to note that most all robust workflow managers such as Nextflow also already include the parts such as "report task completion", "collect task success / failure logs", "report task CPU / memory resource usages", etc.. So that you, the end user, only need to write the parts that implement your data analysis.
Something like proxmox [1] could make wiping everything and restarting a lot easier. There really isn't a huge penalty between bare metal and a VM now so you get the ability to create/deploy/monitor all from a reasonable interface. If the standard clustering stuff isn't enough their datacenter version looks like it is designed more for this. Never used it though. (no ties to proxmox here, just a guy that likes it!)
[1] https://www.proxmox.com/en/products/proxmox-virtual-environm...
So someone re-implemented the Beowulf cluster. I guess the other Slashdot memes are ready to come back.
Hot Grits as a Service
Score: 1 (Insightful)
Don't forget the ultimate /. achievement!
Score: 5 (Troll)
Score: 6 (LLM)
More like -6 =D
I love that you created this account just for this joke. Now we just need Natalie Portman to join.
Can it play Crysis/Doom?
I'm planning to set up something similar (but for compiling code). The difference is that my systems came without storage, so I intend to netboot them, which adds a whole other level of complication. I'm planning to use NixOS, with something like nixos-netboot-serve[0].
https://github.com/DeterminateSystems/nix-netboot-serve
Congratulations on learning about distributed electronic computers. (This is worth tinkering with. This is how people actually get good at HPC.)
Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings.
[Snark] In practice: With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake:
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
Something I've been playing with: what's the cheapest semi-reasonable computer that I could make a toy cluster from? Currently eyeing buying a bunch of Raspberry Pi Zeros but I suspect there are cheaper options (maybe some tiny openwrt thing). Too bad ESP32s don't have an MMU:D
Really depends on the details of your goals, but why not just VMs?
The "problem" with Pi-like devices is that they're usually not very "normal". The process of provisioning is different, IIRC they only "recently" supported booting off something other than the SD Card, and in the case of the Zeros, you'll either be using Wifi or an external USB Ethernet dongle (over USB OTG no less). Sometimes they need specially compiled version of linux, so you're stuck far from mainline (this was a big component of the RPis success) This may be distracting from your goals of learning about clustering.
I suspect the $10 Pi Zero is about as cheap as you'll get though, depending on your personal costs of case + ethernet dongle + USB power supply, etc.
I have a bunch of old Rpi 3B's in a 19" 1RU rack space i use for toy clustering. PoE splitters from a PoE switch to power them which also provides the network uplink. All very neat and self contained and when it comes times to upgrade/replace if they all fail (i have a couple of spares) I can just pull out the 1RU for disposal.
Intel NUC's are probably much better value for money these days but the 3B's were pretty cheap at the time I bought them.
You can get intel NUCs way cheaper if you look around, or the Lenovo mini PCs. Small clusters will never beat a decent CPU but you can probably make a cluster of old mini PCs for less than the price of one.
So obviously it depends on goals; I absolutely agree that up to a surprisingly high point, the best way to actually get the most compute is to go buy the most powerful single machine that will do what you want off eBay or whatever. My goal is the opposite; I want to specifically get my hands dirty figuring out how to actually provision and manage and operate a large number of hosts, even if they're so low resource as to be effectively worthless individually and are still weak clustered. To that end specifically, digikey claims to be willing to sell me a rather large number of pi zero 2w at $15/each - https://www.digikey.com/en/products/detail/raspberry-pi/SC11... - and even cheap used boxes on eBay seem to start at double that. Obviously you need a little bit more to actually run them, but I believe the only thing you actually need is a USB cable and then a single host computer can provide power and boot them over usb.
Just be aware that you’ll be stuck with wifi, or spending far more on ethernet adapters for them. If you want cheap with ethernet there’s other devices supported by Arabian in the same price range which have ethernet. It’s fun for sure to have a huge number of very cheap machines, I have an old arm cluster made of Odroid boards for that purpose.
I actually think I can run network over the USB connections to the head node anyways, though WiFi is also fine.
Actually, you raise a good point: I should spend some time browsing the Armbian supported hardware list...
Kudos for Rscript.