MPI works with OMP and CUDA. This is a comment specific to GPUs about assigning a process to a GPU in a multi-GPU environment. Here is an example:

I have a two computers with 4 GPU's. I want to use both nodes with two GPU's, so I will run mpirun -np 2 ./program args from one machine. Now the question is which two GPU's will be used. Currently, it will round-robin the GPUs (MPI rank 0 gets GPU 0, MPI rank 1 gets GPU 1). I can change this to parse the '--gpu-dev' parameter as an alternative to the round-robin scheme. The current block of code performing the round robin will need to be moved after the parse call, maybe Application::pre_callback. What we don't want is each process getting GPU 0. The '--gpu-dev' flag will need to support multiple values as well, which I think it does?

It is easy to change it to support multiple values, possibly even compile time conditionally. Just give it a vector<int> instead of an int.