> wtf do you need more cores for??? Wat are you doing?
I have a bunch of embarrassingly parallel workloads which I have to run regularly (text analysis, machine learning which is not easily accelerated on a GPU, etc.).
> consider buying a server.
I looked into it; not really an option. Those are vastly more expensive, really loud (I need a quiet workstation), have significantly lower boost clocks (I also need single-thread performance), and are either really hard or even impossible to actually buy in my country through normal retail channels.
Alas you can't do that for every workload, and even for those where you can that brings extra complexity. On a single machine I can just replace an `.iter()` into a `.par_iter()` in my Rust programs and boom, I parallelized my workload across every core with only a single line change. It's not that simple with multiple machines. (I might do that one day, but I'm not there yet.)
> On a single machine I can just replace an `.iter()` into a `.par_iter()` in my Rust programs and boom, I parallelized my workload across every core with only a single line change.
If you’re using Rayon for anything you expect to benefit from a lot of cores, my experiences have been very negative in the past. Rayon is often extremely inefficient once you allow it to use more than a handful of cores.[0]
If your unit of work is absolutely massive, you might not be as affected by Rayon’s scheduler, but it makes me sad that this issue still hasn’t been resolved. My comment on that thread (documenting just one instance where I ran into this issue) was almost 4 years ago! And the issue itself is almost a year older than that. Sure, I could try to contribute a general solution, but it’s just easier to stop using Rayon.
I no longer consider Rayon to be an advantage for Rust. There are plenty of good ways to wire up a parallel work pool, but they do require more than a one line change.
Gotcha. But FYI I wrote a work farmer years ago and it wasn't difficult. You just manually started a program over there and started the farmer over here and that was all. It could use every machine in the office. These days there may be better solutions than hand-rolling it, never used it but <https://en.wikipedia.org/wiki/GNU_parallel> might work. Also see if beowulf software has anything to make it easier. Good luck!
I have a bunch of embarrassingly parallel workloads which I have to run regularly (text analysis, machine learning which is not easily accelerated on a GPU, etc.).
> consider buying a server.
I looked into it; not really an option. Those are vastly more expensive, really loud (I need a quiet workstation), have significantly lower boost clocks (I also need single-thread performance), and are either really hard or even impossible to actually buy in my country through normal retail channels.