There are two things every programmer of multithreaded applications knows:
1. Logging is the simplest form of debugging for such systems.
2. The act of logging triggers context switches which changes the execution which in turn means the application behaves differently under debugging conditions than it does under normal conditions.
Often the application of #1 results in #2 meaning the “bug” can go away or change in nature to the point where the original bug is no longer obvious. You can assume that you have the bug fixed yet as soon as you remove/disable the debug code and the bug can return or worse, uncover a completely new one.
Part of the reason for this is that writing to a log, socket or other place to store the log will cause the kernel to execute a context switch. This means that different threads will execute in a different order under debug conditions…this is what I call “the observer effect”: a watched program executes differently than one running in release mode. This is even more noticeable when running it under GDB or Valgrind for example. There are two ways to deal with this:
