SimGrid  3.18
Versatile Simulation of Distributed Systems
SMPI: Simulate real MPI applications

Programming environment for the simulation of MPI applications.

SMPI enables the study of MPI application by emulating them on top of the SimGrid simulator. This is particularly interesting to study existing MPI applications within the comfort of the simulator. The SMPI reference article is available at https://hal.inria.fr/hal-01415484. You should also read the SMPI introductory slides.

Our goal is to enable the study of unmodified MPI applications. Some constructs and features are still missing, but we can probably add them on demand. If you already used MPI before, SMPI should sound very familiar to you: Use smpicc instead of mpicc, and smpirun instead of mpirun. The main difference is that smpirun takes a virtual platform as extra parameter (see Describing the virtual platform).

If you are new to MPI, you should first take our online SMPI CourseWare. It consists in several projects that progressively introduce the MPI concepts. It proposes to use SimGrid and SMPI to run the experiments, but the learning objectives are centered on MPI itself.

For further scalability, you may modify your code to speed up your studies or save memory space. Maximal simulation accuracy requires some specific care from you.

Using SMPI

Compiling your code

If your application is in C, then simply use smpicc as a compiler just like you use mpicc with other MPI implementations. This script still calls your default compiler (gcc, clang, ...) and adds the right compilation flags along the way. If your application is in C++, Fortran 77 or Fortran 90, use respectively smpicxx, smpiff or smpif90.

Executing your code on the simulator

Use the smpirun script as follows for that:

smpirun -hostfile my_hostfile.txt -platform my_platform.xml ./program -blah

smpirun accepts other parameters, such as -np if you don't want to use all the hosts defined in the hostfile, -map to display on which host each rank gets mapped of -trace to activate the tracing during the simulation. You can get the full list by running

smpirun -help

Simulating collective operations

MPI collective operations are crucial to the performance of MPI applications and must be carefully optimized according to many parameters. Every existing implementation provides several algorithms for each collective operation, and selects by default the best suited one, depending on the sizes sent, the number of nodes, the communicator, or the communication library being used. These decisions are based on empirical results and theoretical complexity estimation, and are very different between MPI implementations. In most cases, the users can also manually tune the algorithm used for each collective operation.

SMPI can simulate the behavior of several MPI implementations: OpenMPI, MPICH, STAR-MPI, and MVAPICH2. For that, it provides 115 collective algorithms and several selector algorithms, that were collected directly in the source code of the targeted MPI implementations.

You can switch the automatic selector through the smpi/coll-selector configuration item. Possible values:

Available algorithms

You can also pick the algorithm used for each collective with the corresponding configuration item. For example, to use the pairwise alltoall algorithm, one should add –cfg=smpi/alltoall:pair to the line. This will override the selector (if any) for this algorithm. It means that the selected algorithm will be used

Warning: Some collective may require specific conditions to be executed correctly (for instance having a communicator with a power of two number of nodes only), which are currently not enforced by Simgrid. Some crashes can be expected while trying these algorithms with unusual sizes/parameters

MPI_Alltoall

Most of these are best described in STAR-MPI

MPI_Alltoallv

MPI_Gather

MPI_Barrier

MPI_Scatter

MPI_Reduce

MPI_Allreduce

MPI_Reduce_scatter

MPI_Allgather

MPI_Allgatherv

MPI_Bcast

Automatic evaluation

(Warning: This is still very experimental)

An automatic version is available for each collective (or even as a selector). This specific version will loop over all other implemented algorithm for this particular collective, and apply them while benchmarking the time taken for each process. It will then output the quickest for each process, and the global quickest. This is still unstable, and a few algorithms which need specific number of nodes may crash.

Adding an algorithm

To add a new algorithm, one should check in the src/smpi/colls folder how other algorithms are coded. Using plain MPI code inside Simgrid can't be done, so algorithms have to be changed to use smpi version of the calls instead (MPI_Send will become smpi_mpi_send). Some functions may have different signatures than their MPI counterpart, please check the other algorithms or contact us using SimGrid developers mailing list.

Example: adding a "pair" version of the Alltoall collective.

Tracing of internal communications

By default, the collective operations are traced as a unique operation because tracing all point-to-point communications composing them could result in overloaded, hard to interpret traces. If you want to debug and compare collective algorithms, you should set the tracing/smpi/internals configuration item to 1 instead of 0.

Here are examples of two alltoall collective algorithms runs on 16 nodes, the first one with a ring algorithm, the second with a pairwise one:


What can run within SMPI?

You can run unmodified MPI applications (both C/C++ and Fortran) within SMPI, provided that you only use MPI calls that we implemented. Global variables should be handled correctly on Linux systems.

MPI coverage of SMPI

Our coverage of the interface is very decent, but still incomplete; Given the size of the MPI standard, we may well never manage to implement absolutely all existing primitives. Currently, we have almost no support for I/O primitives, but we still pass a very large amount of the MPICH coverage tests.

The full list of not yet implemented functions is documented in the file include/smpi/smpi.h, between two lines containing the FIXME marker. If you really miss a feature, please get in touch with us: we can guide you though the SimGrid code to help you implementing it, and we'd glad to integrate your contribution to the main project afterward.

Privatization of global variables

Concerning the globals, the problem comes from the fact that usually, MPI processes run as real UNIX processes while they are all folded into threads of a unique system process in SMPI. Global variables are usually private to each MPI process while they become shared between the processes in SMPI. The problem and some potential solutions are discussed in this article: "Automatic Handling of Global Variables for Multi-threaded MPI Programs", available at http://charm.cs.illinois.edu/newPapers/11-23/paper.pdf (note that this article does not deal with SMPI but with a competing solution called AMPI that suffers of the same issue). This point used to be problematic in SimGrid, but the problem should now be handled automatically on Linux.

Older versions of SimGrid came with a script that automatically privatized the globals through static analysis of the source code. But our implementation was not robust enough to be used in production, so it was removed at some point. Currently, SMPI comes with two privatization mechanisms that you can select at runtime. At the time of writing (v3.18), the dlopen approach is considered to be very fast (it's used by default) while the mmap approach is considered to be rather slow but very robust.

With the mmap approach, SMPI duplicates and dynamically switch the .data and .bss segments of the ELF process when switching the MPI ranks. This allows each ranks to have its own copy of the global variables. No copy actually occures as this mechanism uses mmap for efficiency. This mechanism is considered to be very robust on all systems supporting mmap (Linux and most BSDs). Its performance is questionable since each context switch between MPI ranks induces several syscalls to change the mmap that redirects the .data and .bss segments to the copies of the new rank. The code will also be copied several times in memory, inducing a slight increase of memory occupation.

Another limitation is that SMPI only accounts for global variables defined in the executable. If the processes use external global variables from dynamic libraries, they won't be switched correctly. The easiest way to solve this is to statically link against the library with these globals. This way, each MPI rank will get its own copy of these libraries. Of course you should never statically link against the SimGrid library itself.

With the dlopen approach, SMPI loads several copies of the same executable in memory as if it were a library, so that the global variables get naturally duplicated. It first requires the executable to be compiled as a relocatable binary, which is less common for programs than for libraries. But most distributions are now compiled this way for security reason as it allows to randomize the address space layout. It should thus be safe to compile most (any?) program this way. The second trick is that the dynamic linker refuses to link the exact same file several times, be it a library or a relocatable executable. It makes perfectly sense in the general case, but we need to circumvent this rule of thumb in our case. To that extend, the binary is copied in a temporary file before being re-linked against.

This approach greatly speeds up the context switching, down to about 40 CPU cycles with our raw contextes, instead of requesting several syscalls with the mmap approach. Another advantage is that it permits to run the SMPI contexts in parallel, which is obviously not possible with the mmap approach. It was tricky to implement, but we are not aware of any flaws, so smpirun activates it by default.

In the future, it may be possible to further reduce the memory and disk consumption. It seems that we could punch holes in the files before dl-loading them to remove the code and constants, and mmap these area onto a unique copy. If done correctly, this would reduce the disk- and memory- usage to the bare minimum, and would also reduce the pressure on the CPU instruction cache.
Also, currently, only the binary is copied and dlopen-ed for each MPI rank. We could probably extend this to external dependencies, but for now, any external dependencies must be statically linked into your application. As usual, simgrid itself shall never be statically linked in your app. You don't want to give a copy of SimGrid to each MPI rank: that's ways too much for them to deal with.

Adapting your MPI code for further scalability

As detailed in the reference article (available at http://hal.inria.fr/hal-01415484), you may want to adapt your code to improve the simulation performance. But these tricks may seriously hinder the result quality (or even prevent the app to run) if used wrongly. We assume that if you want to simulate an HPC application, you know what you are doing. Don't prove us wrong!

Reducing your memory footprint

If you get short on memory (the whole app is executed on a single node when simulated), you should have a look at the SMPI_SHARED_MALLOC and SMPI_SHARED_FREE macros. It allows to share memory areas between processes: The purpose of these macro is that the same line malloc on each process will point to the exact same memory area. So if you have a malloc of 2M and you have 16 processes, this macro will change your memory consumption from 2M*16 to 2M only. Only one block for all processes.

If your program is ok with a block containing garbage value because all processes write and read to the same place without any kind of coordination, then this macro can dramatically shrink your memory consumption. For example, that will be very beneficial to a matrix multiplication code, as all blocks will be stored on the same area. Of course, the resulting computations will useless, but you can still study the application behavior this way.

Naturally, this won't work if your code is data-dependent. For example, a Jacobi iterative computation depends on the result computed by the code to detect convergence conditions, so turning them into garbage by sharing the same memory area between processes does not seem very wise. You cannot use the SMPI_SHARED_MALLOC macro in this case, sorry.

This feature is demoed by the example file examples/smpi/NAS/dt.c

Toward faster simulations

If your application is too slow, try using SMPI_SAMPLE_LOCAL, SMPI_SAMPLE_GLOBAL and friends to indicate which computation loops can be sampled. Some of the loop iterations will be executed to measure their duration, and this duration will be used for the subsequent iterations. These samples are done per processor with SMPI_SAMPLE_LOCAL, and shared between all processors with SMPI_SAMPLE_GLOBAL. Of course, none of this will work if the execution time of your loop iteration are not stable.

This feature is demoed by the example file examples/smpi/NAS/ep.c

Ensuring accurate simulations

Out of the box, SimGrid may give you fairly accurate results, but there is a plenty of factors that could go wrong and make your results inaccurate or even plainly wrong. Actually, you can only get accurate results of a nicely built model, including both the system hardware and your application. Such models are hard to pass over and reuse in other settings, because elements that are not relevant to an application (say, the latency of point-to-point communications, collective operation implementation details or CPU-network interaction) may be irrelevant to another application. The dream of the perfect model, encompassing every aspects is only a chimera, as the only perfect model of the reality is the reality. If you go for simulation, then you have to ignore some irrelevant aspects of the reality, but which aspects are irrelevant is actually application-dependent...

The only way to assess whether your settings provide accurate results is to double-check these results. If possible, you should first run the same experiment in simulation and in real life, gathering as much information as you can. Try to understand the discrepancies in the results that you observe between both settings (visualization can be precious for that). Then, try to modify your model (of the platform, of the collective operations) to reduce the most preeminent differences.

If the discrepancies come from the computing time, try adapting the smpi/host-speed: reduce it if your simulation runs faster than in reality. If the error come from the communication, then you need to fiddle with your platform file.

Be inventive in your modeling. Don't be afraid if the names given by SimGrid does not match the real names: we got very good results by modeling multicore/GPU machines with a set of separate hosts interconnected with very fast networks (but don't trust your model because it has the right names in the right place either).

Finally, you may want to check this article on the classical pitfalls in modeling distributed systems.

Troubleshooting with SMPI

My ./configure refuses to use smpicc

Alas, some building infrastructures cannot use smpicc as a project compiler, and your ./configure may report that the compiler is not functional. If this happens, define the SMPI_PRETEND_CC environment variable before running the configuration.

SMPI_PRETEND_CC=1 ./configure # here come the configure parameters
make
Warning

Make sure that SMPI_PRETEND_CC is only set when calling ./configure, not during the actual compilation. With that variable, smpicc does not do anything, to not hurt the ./configure feelings. But you need smpicc do actually do something to get your application compiled.