SimGrid  3.19.1 Versatile Simulation of Distributed Systems

This document is the FAQ of the MSG interface. Some entries are a bit aging and it should be refreshed at some point.

# I'm new to SimGrid. I have some questions. Where should I start?

You are at the right place... To understand what you can do or cannot do with SimGrid, you should read the tutorial slides from the SimGrid's website. You may find more uptodate material on the blog of Martin Quinson.

Another great source of inspiration can be found in the S4U examples.

If you are stuck at any point and if this FAQ cannot help you, please drop us a mail to the user mailing list: simgr.nosp@m.id-u.nosp@m.ser@l.nosp@m.ists.nosp@m..gfor.nosp@m.ge.i.nosp@m.nria..nosp@m.fr.

## What is the difference between MSG and SimDag? Do they serve the same purpose?

It depend on how you define "purpose", I guess ;)

They all allow you to build a prototype of application which you can run within the simulator afterward. They all share the same simulation kernel, which is the core of the SimGrid project. They differ by the way you express your application.

With SimDag, you express your code as a collection of interdependent parallel tasks. So, in this model, applications can be seen as a DAG of tasks. This is the interface of choice for people wanting to port old code designed for SimGrid v1 or v2 to the framework current version.

With MSG, your application is seen as a set of communicating processes, exchanging data by the way of messages and performing computation on their own.

## Visualizing and analyzing the results

It is sometime convenient to "see" how the agents are behaving. If you like colors, you can use tools/MSG_visualization/colorize.pl  as a filter to your MSG outputs. It works directly with INFO. Beware, INFO() prints on stderr. Do not forget to redirect if you want to filter (e.g. with bash):

./msg_test small_platform.xml small_deployment.xml 2>&1 | ../../tools/MSG_visualization/colorize.pl


We also have a more graphical output. Have a look at section Configuring the tracing subsystem.

## Argh! Do I really have to code in C?

We provide Java bindings of the MSG interface, which is the main SimGrid user API.

Moreover If you use C++, you should be able to use the SimGrid library as a standard C library and everything should work fine (simply link against this library; recompiling SimGrid with a C++ compiler won't work and it wouldn't help if you could).

For now, we do not feel a real demand for any other language. But if you think there is one, please speak up!

# Feature related questions

You'll find in this section a few "Missing In Action" features. Many people have asked about it and we have given hints on how to simply do it with MSG. Feel free to contribute...

## MSG features

### I want some more complex MSG examples!

Many people have come to ask me a more complex example and each time, they have realized afterward that the basics were in the previous three examples.

Of course they have often been needing more complex functions like MSG_process_suspend(), MSG_process_resume() and MSG_process_isSuspended() (to perform synchronization), or MSG_task_Iprobe() and MSG_process_sleep() (to avoid blocking receptions), or even MSG_process_create() (to design asynchronous communications or computations). But the examples are sufficient to start.

We know. We should add some more examples, but not really some more complex ones... We should add some examples that illustrate some other functionalists (like how to simply encode asynchronous communications, RPC, process migrations, thread synchronization, ...) and we will do it when we will have a little bit more time. We have tried to document the examples so that they are understandable. Tell us if something is not clear and once again feel free to participate! :)

### Missing in action: MSG Task duplication/replication

There is no task duplication in MSG. When you create a task, you can process it or send it somewhere else. As soon as a process has sent this task, he doesn't have this task anymore. It's gone. The receiver process has got the task. However, you could decide upon receiving to create a "copy" of a task but you have to handle by yourself the semantic associated to this "duplication".

As we already told, we prefer keeping the API as simple as possible. This kind of feature is rather easy to implement by users and the semantic you associate really depends on people. Having a generic* task duplication mechanism is not that trivial (in particular because of the data field). That is why I would recommend that you write it by yourself even if I can give you advice on how to do it.

You could use a dictionary (xbt_dict_t) of dynars (xbt_dynar_t). If you still don't see how to do it, please come back to us...

### How to synchronize my user processes?

It depends on why you want to synchronize them. If you just want to have a shared state between your processes, then you probably don't need to do anything. User processes never get forcefully interrupted in SimGrid (unless you explicitly request the parallel execution of user processes – see Running user code in parallel).

Even if several processes are executed at the exact same time within the simulation, they are linearized in reality by default: one process always run in an exclusive manner, atomic, uninterrupted until it does a simcall (until it ask a service from the simulation kernel). This is surprising at first, but things are much easier this way, both for the user (who don't have to protect her shared data) and for the kernel (that avoid many synchronization issues too). Processes are executed concurrently in the simulated realm, but you don't need to bother with this in the real realm.

If you really need to synchronize your processes (because it's what you are studying or to create an atomic section that spans over several simcalls), you obviously cannot use regular synchronization mechanisms (pthread_mutexes in C or the synchronized keyword in Java). This is because the SimGrid kernel locks all processes and unlock them one after the other when they are supposed to run, until they give the control back in their simcall. If one of them gets locked by the OS before returning the control to the kernel, that's definitively a deadlock.

Instead, you should use the synchronization mechanism provided by the simulation kernel. This could with a SimGrid mutex, a SimGrid condition variables or a SimGrid semaphore, as described in Explicit Synchronization Functions (in Java, only semaphores are available). But actually, many synchronization patterns can be encoded with communication on mailboxes. Typically, if you need one process to notify another one, you could use a condition variable or a semphore, but sending a message to a specific mailbox does the trick in most cases.

### Where is the get_host_load function hidden in MSG?

There is no such thing because its semantic wouldn't be really clear. Of course, it is something about the amount of host throughput, but there is as many definition of "host load" as people asking for this function. First, you have to remember that resource availability may vary over time, which make any load notion harder to define.

It may be instantaneous value or an average one. Moreover it may be only the power of the computer, or may take the background load into account, or may even take the currently running tasks into account. In some SURF models, communications have an influence on computational power. Should it be taken into account too?

First of all, it's near to impossible to predict the load beforehand in the simulator since it depends on too much parameters (background load variation, bandwidth sharing algorithmic complexity) some of them even being not known beforehand (other task starting at the same time). So, getting this information is really hard (just like in real life). It's not just that we want MSG to be as painful as real life. But as it is in some way realistic, we face some of the same problems as we would face in real life.

How would you do it for real? The most common option is to use something like NWS that performs active probes. The best solution is probably to do the same within MSG, as in next code snippet. It is very close from what you would have to do out of the simulator, and thus gives you information that you could also get in real settings to not hinder the realism of your simulation.

double date = MSG_get_clock();
date = MSG_get_clock() - date;
return (0.001/date);
}

Of course, it may not match your personal definition of "host load". In this case, please detail what you mean on the mailing list, and we will extend this FAQ section to fit your taste if possible.

### How can I get the *real* communication time?

Communications are synchronous and thus if you simply get the time before and after a communication, you'll only get the transmission time and the time spent to really communicate (it will also take into account the time spent waiting for the other party to be ready). However, getting the real communication time is not really hard either. The following solution is a good starting point.

int sender()
{
calloc(1,sizeof(double)));
XBT_INFO("Send completed");
return 0;
}
{
double time1,time2;
time1 = MSG_get_clock();
time2 = MSG_get_clock();
XBT_INFO("Communication time : \"%f\" ", time2-time1);
return 0;
}

## SimDag related questions

### Implementing communication delays between tasks.

A classic question of SimDag newcomers is about how to express a communication delay between tasks. The thing is that in SimDag, both computation and communication are seen as tasks. So, if you want to model a data dependency between two DAG tasks t1 and t2, you have to create 3 SD_tasks: t1, t2 and c and add dependencies in the following way:

This way task t2 cannot start before the termination of communication c which in turn cannot start before t1 ends.

When creating task c, you have to associate an amount of data (in bytes) corresponding to what has to be sent by t1 to t2.

Finally to schedule the communication task c, you have to build a list comprising the workstations on which t1 and t2 are scheduled (w1 and w2 for example) and build a communication matrix that should look like [0;amount ; 0; 0].

### How to implement a distributed dynamic scheduler of DAGs.

Distributed is somehow "contagious". If you start making distributed decisions, there is no way to handle DAGs directly anymore (unless I am missing something). You have to encode your DAGs in term of communicating process to make the whole scheduling process distributed. Here is an example of how you could do that. Assume T1 has to be done before T2.

int your_agent(int argc, char *argv[] {
...
...
while(1) {
...
...
else {
/* do something else */
}
}
}

If you decide that the distributed part is not that much important and that DAG is really the level of abstraction you want to work with, then you should give a try to SimDag: Legacy handling of DAG algorithms.

## Generic features

### Is there a native support for batch schedulers in SimGrid?

No, there is no native support for batch schedulers and none is planned because this is a very specific need (and doing it in a generic way is thus very hard). However some people have implemented their own batch schedulers. Vincent Garonne wrote one during his PhD and put his code in the contrib directory of our SVN so that other can keep working on it. You may find inspiring ideas in it.

### I need a checkpointing thing

Actually, it depends on whether you want to checkpoint the simulation, or to simulate checkpoints.

The first one could help if your simulation is a long standing process you want to keep running even on hardware issues. It could also help to rewind the simulation by jumping sometimes on an old checkpoint to cancel recent calculations.
Unfortunately, such thing will probably never exist in SG. One would have to duplicate all data structures because doing a rewind at the simulator level is very very hard (not talking about the malloc free operations that might have been done in between). Instead, you may be interested in the Libckpt library (http://www.cs.utk.edu/~plank/plank/www/libckpt.html). This is the checkpointing solution used in the condor project, for example. It makes it easy to create checkpoints (at the OS level, creating something like core files), and rerunning them on need.

If you want to simulate checkpoints instead, it means that you want the state of an executing task (in particular, the progress made towards completion) to be saved somewhere. So if a host (and the task executing on it) fails (cf. MSG_HOST_FAILURE), then the task can be restarted from the last checkpoint.
Actually, such a thing does not exist in SimGrid either, but it's just because we don't think it is fundamental and it may be done in the user code at relatively low cost. You could for example use a watcher that periodically get the remaining amount of things to do (using MSG_task_get_remaining_computation()), or fragment the task in smaller subtasks.

## Platform building and Dynamic resources

### Where can I find SimGrid platform files?

There are several little examples in the archive, in the examples/msg directory. From time to time, we are asked for other files, but we don't have much at hand right now.

You should refer to the Platform Description Archive (http://pda.gforge.inria.fr) project to see the other platform file we have available, as well as the Simulacrum simulator, meant to generate SimGrid platforms using all classical generation algorithms.

### How can I automatically map an existing platform?

We are working on a project called ALNeM (Application-Level Network Mapper) which goal is to automatically discover the topology of an existing network. Its output will be a platform description file following the SimGrid syntax, so everybody will get the ability to map their own lab network (and contribute them to the catalog project). This tool is not ready yet, but it move quite fast forward. Just stay tuned.

### Generating synthetic but realistic platforms

The third possibility to get a platform file (after manual or automatic mapping of real platforms) is to generate synthetic platforms. Getting a realistic result is not a trivial task, and moreover, nobody is really able to define what "realistic" means when speaking of topology files. You can find some more thoughts on this topic in these slides.

If you are looking for an actual tool, there we have a little tool to annotate Tiers-generated topologies. This perl-script is in tools/platform_generation/ directory of the SVN. Dinda et Al. released a very comparable tool, and called it GridG.

The specified computing power will be available to up to 6 sequential tasks without sharing. If more tasks are placed on this host, the resource will be shared accordingly. For example, if you schedule 12 tasks on the host, each will get half of the computing power. Please note that although sound, this model were never scientifically assessed. Please keep this fact in mind when using it.

### Using random variable for the resource power or availability

The best way to model the resouce power using a random variable is to use an availability trace that is directed by a probability distribution. This can be done using the function tmgr_trace_generator_value() below. The date and value generators is created with one of tmgr_event_generator_new_uniform(), tmgr_event_generator_new_exponential() or tmgr_event_generator_new_weibull() (if you need other generators, adding them to src/surf/trace_mgr.c should be quite trivial and your patch will be welcomed). Once your trace is created, you have to connect it to the resource with the function sg_platf_new_trace_connect().

That the process is very similar if you want to model the resource availability with a random variable (deciding whether it's on/off instead of deciding its speed) using the function tmgr_trace_generator_state() or tmgr_trace_generator_avail_unavail() instead of tmgr_trace_generator_value().

Unfortunately, all this is currently lacking a proper documentation, and there is even no proper example of use. You'll thus have to check the header file include/simgrid/platf.h and experiment a bit by yourself. The following code should be a good starting point, and contributing a little clean example would be a good way to help the SimGrid project.

tmgr_trace_generator_value("mytrace",tmgr_event_generator_new_exponential(.5)
tmgr_event_generator_new_uniform(100000,9999999));
sg_platf_trace_connect_cbarg_t myconnect = SG_PLATF_TRACE_CONNECT_INITIALIZER;
myconnect.kind = SURF_TRACE_CONNECT_KIND_BANDWIDTH;
myconnect.trace = "mytrace";

# Troubleshooting

## The feature X stopped to work after my last update

I guess that you want to read the ChangeLog file, that always contains all the information that could be important to the users during the upgrade. Actually, you may want to read it (alongside with the NEWS file that highlights the most important changes) even before you upgrade your copy of SimGrid, too.

We do our best to maintain the backward compatibility, but we sometimes have to fix the things that are too broken. If we happen to kill a feature that you were using, we are sorry. We think that you should update to the new way of doing things, but if you can't afford it, that's ok. Just stick to the last version that were working for you, and have a pleasant day.

## SimGrid compilation and installation problems

### cmake fails!

We know only one reason for the configure to fail:

• You are using a broken build environment
Try updating your cmake version. If symptom is that the configury magic complains about gcc not being able to build executables, you are probably missing the libc6-dev package. Damn Ubuntu.

If you experience other kind of issue, please get in touch with us. We are always interested in improving our portability to new systems.

### Dude! "ctest" fails on my machine!

Don't assume we never run this target, because we do. Check http://cdash.inria.fr/CDash/index.php?project=Simgrid (click on previous if there is no result for today: results are produced only by 11am, French time) and https://buildd.debian.org/status/logs.php?pkg=simgrid if you don't believe us.

If it's failing on your machine in a way not experienced by the autobuilders above, please drop us a mail on the mailing list so that we can check it out. Make sure to read So I've found a bug in SimGrid. How to report it? before you do so.

## User code compilation problems

### "gcc: _simgrid_this_log_category_does_not_exist__??? undeclared (first use in this function)"

This is because you are using the log mechanism, but you didn't created any default category in this file. You should refer to Logging support for all the details, but you simply forgot to call one of XBT_LOG_NEW_DEFAULT_CATEGORY() or XBT_LOG_NEW_DEFAULT_SUBCATEGORY().

### "gcc: undefined reference to pthread_key_create"

This indicates that one of the library SimGrid depends on (libpthread here) was missing on the linking command line. Dependencies of libsimgrid are expressed directly in the dynamic library, so it's quite impossible that you see this message when doing dynamic linking.

If you compile your code statically (and if you use a pthread version of SimGrid), you must absolutely specify -lpthread on the linker command line. As usual, this should come after -lsimgrid on this command line.

## Runtime error messages

### I'm told that my XML files are too old.

The format of the XML platform description files is sometimes improved. For example, we decided to change the units used in SimGrid from MBytes, MFlops and seconds to Bytes, Flops and seconds to ease people exchanging small messages. We also reworked the route descriptions to allow more compact descriptions.

That is why the XML files are versionned using the 'version' attribute of the root tag. Currently, it should read:

  <platform version="4">


If your files are too old, you can use the simgrid_update_xml.pl script which can be found in the tools directory of the archive.

## Debugging SMPI applications

In order to debug SMPI programs, you can use the following options:

• -wrapper 'gdb –args': this option is used to use a wrapper in order to call the SMPI process. Good candidates for this options are "gdb --args", "valgrind", "rr record", "strace", etc;
• -foreground: this options gives the debugger access to the terminal which is needed in order to use an interactive debugger.

Both options are needed in order to run the SMPI process under GDB.

## Valgrind-related and other debugger issues

If you don't, you really should use valgrind to debug your code, it's almost magic.

### Valgrind spits tons of errors about backtraces!

It may happen that valgrind, the memory debugger beloved by any decent C programmer, spits tons of warnings like the following :

==8414== Conditional jump or move depends on uninitialised value(s)
==8414==    at 0x400882D: (within /lib/ld-2.3.6.so)
==8414==    by 0x414EDE9: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==8414==    by 0x400B105: (within /lib/ld-2.3.6.so)
==8414==    by 0x414F937: _dl_open (in /lib/tls/i686/cmov/libc-2.3.6.so)
==8414==    by 0x4150F4C: (within /lib/tls/i686/cmov/libc-2.3.6.so)
==8414==    by 0x400B105: (within /lib/ld-2.3.6.so)
==8414==    by 0x415102D: __libc_dlopen_mode (in /lib/tls/i686/cmov/libc-2.3.6.so)
==8414==    by 0x412D6B9: backtrace (in /lib/tls/i686/cmov/libc-2.3.6.so)
==8414==    by 0x8076446: xbt_dictelm_get_ext (dict_elm.c:714)
==8414==    by 0x80764C1: xbt_dictelm_get (dict_elm.c:732)
==8414==    by 0x8079010: xbt_cfg_register (config.c:208)
==8414==    by 0x806821B: MSG_config (msg_config.c:42)


This problem is somewhere in the libc when using the backtraces and there is very few things we can do ourselves to fix it. Instead, here is how to tell valgrind to ignore the error. Add the following to your ~/.valgrind.supp (or create this file on need). Make sure to change the obj line according to your personnal mileage (change 2.3.6 to the actual version you are using, which you can retrieve with a simple "ls /lib/ld*.so").

{
Memcheck:Cond
obj:/lib/ld-2.3.6.so
fun:dl_open_worker
fun:_dl_open
fun:do_dlopen
fun:dlerror_run
fun:__libc_dlopen_mode
}

Then, you have to specify valgrind to use this suppression file by passing the –suppressions=$HOME/.valgrind.supp option on the command line. You can also add the following to your ~/.bashrc so that it gets passed automatically. Actually, it passes a bit more options to valgrind, and this happen to be my personnal settings. Check the valgrind documentation for more information. export VALGRIND_OPTS="--leak-check=yes --leak-resolution=high --num-callers=40 --tool=memcheck --suppressions=$HOME/.valgrind.supp"

### Truncated backtraces

When debugging SimGrid, it's easier to pass the –disable-compiler-optimization flag to the configure if valgrind or gdb get fooled by the optimization done by the compiler. But you should remove these flag when everything works before going in production (before launching your 1252135 experiments), or everything will run only one half of the true SimGrid potential.

## There is a deadlock in my code!!!

Unfortunately, we cannot debug every code written in SimGrid. We furthermore believe that the framework provides ways enough information to debug such information yourself. If the textual output is not enough, Make sure to check the Visualizing and analyzing the results FAQ entry to see how to get a graphical one.

Now, if you come up with a really simple example that deadlocks and you're absolutely convinced that it should not, you can ask on the list. Just be aware that you'll be severely punished if the mistake is on your side... We have plenty of FAQ entries to redact and new features to implement for the impenitents! ;)

## I get weird timings when I play with the latencies.

OK, first of all, remember that units should be Bytes, Flops and Seconds. If you don't use such units, some SimGrid constants (e.g. the SG_TCP_CTE_GAMMA constant used in most network models) won't have the right unit and you'll end up with weird results.

Here is what happens with a single transfer of size L on a link (bw,lat) when nothing else happens.

0-----lat--------------------------------------------------t
|-----|**** real_bw =min(bw,SG_TCP_CTE_GAMMA/(2*lat)) *****|


In more complex situations, this min is the solution of a complex max-min linear system. Have a look here and read the two threads "Bug in SURF?" and "Surf bug not fixed?". You'll have a few other examples of such computations. You can also read "A Network Model for Simulation of Grid Application" by Henri Casanova and Loris Marchal to have all the details. The fact that the real_bw is smaller than bw is easy to understand. The fact that real_bw is smaller than SG_TCP_CTE_GAMMA/(2*lat) is due to the window-based congestion mechanism of TCP. With TCP, you can't exploit your huge network capacity if you don't have a good round-trip-time because of the acks...

Anyway, what you get is t=lat + L/min(bw,SG_TCP_CTE_GAMMA/(2*lat)).

if I you set (bw,lat)=(100 000 000, 0.00001), you get t =  1.00001 (you fully


use your link) if I you set (bw,lat)=(100 000 000, 0.0001), you get t = 1.0001 (you're on the limit) if I you set (bw,lat)=(100 000 000, 0.001), you get t = 10.001 (ouch!)

This bound on the effective bandwidth of a flow is not the only thing that may make your result be unexpected. For example, two flows competing on a saturated link receive an amount of bandwidth inversely proportional to their round trip time.

## So I've found a bug in SimGrid. How to report it?

We do our best to make sure to hammer away any bugs of SimGrid, but this is still an academic project so please be patient if/when you find bugs in it. If you do, the best solution is to drop an email either on the simgrid-user or the simgrid-devel mailing list and explain us about the issue. You can also decide to open a formal bug report using the relevant interface. You need to login on the server to get the ability to submit bugs.

We will do our best to solve any problem repported, but you need to help us finding the issue. Just telling "it segfault" isn't enough. Telling "It segfaults when running the attached simulator" doesn't really help either. You may find the following article interesting to see how to repport informative bug repports: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html (it is not SimGrid specific at all, but it's full of good advices).