I’m currently trying to get a piece of c++ code from my Macbook to run on the cluster, which means I have to compile it for linux first. However, there seem to be lots of ways to go about that and I’m not familiar with any of them. So therefore my question: are there any people here who do this and can recommend a method?
So far I’ve come across the following options:
compile the code on the cluster itself (but I haven’t figured out how to do this properly?)
use a virtual machine to crosscompile (there are many though)
use docker to crosscompile
am I missing something?
I’m interested in which methods people are using and what the benefits or downsides are. If you don’t want to type but have some other resource for me it is also much appreciated.
Technically you don’t even need to use any sort of virtualization for cross-compilation, you need only to have a compiler for a proper architecture installed, for example this Stackoverflow answer. However, you will stuck into a lot of issues with linking, because most probably set of libraries on your PC will be different from the set of libraries on the cluster, and I doubt it is easier to solve than compiling on the cluster.
I would recommend to figure out how to compile it on the cluster: seems like compiler is installed there, but it is ancient, it is up to you to figure out what goes wrong, or at least ask the question telling what’s going wrong precisely.
If this does not work, you can try to cross-compile on the Mac for x86-64-linux architecture (have no Mac and not much expertise to help). You have to ensure that you link everything statically, otherwise you will probably find a lot of linking issues when you move that binary to the cluster and it doesn’t find necessary libraries.
In the worst case, our clusters (at least hpc05) have Singularity installed, and you can use it to package the whole environment on the local machine and transfer your code with environment to cluster.
Thank you for taking the time to respond. In the end I chose to use docker for cross compilation since the compiler on the cluster is quite old. Indeed there were some issues with linking libraries but by doing everything statically that was solved in the end!
I can recommend docker to anyone coming here with the same issue, it’s quite easy to set up and there is good documentation out there.
As a bonus point: I don’t remember how uniform our clusters are, but if they are one can try to resolve supported GCC optimization flags and build binaries optimized for the processors of the cluster. This might give you some performance points (I am not sure how much though).