LTTng User-space Tracing Design

A user-space application may contain static instrumentation, and be linked with an associated runtime library, in order to produce an execution trace. User space Tracepoints and Markers, analogous to Kernel Tracepoints and Markers, define program locations and arguments provided. Probes may then be connected to each Tracepoint and Marker before they are activated. The probes will typically call LTTng user-space event writing functions.

When the instrumented application starts, a pipe is opened to allow external tracing control, asynchronous notification is requested when commands arrive in the pipe, and a signal handler is installed for SIGIO (or a carefully chosen chainable signal number). Every time such signal is received, the runtime library checks for commands received on the external tracing control pipe. The recognized commands are:

  • list the available Markers and Tracepoints;
  • dynamically load a library (presumably containing probes to connect);
  • connect a probe to a Tracepoint or Marker;
  • activate a Tracepoint or Marker;
  • initialize the tracing buffers;
  • flush the tracing buffers;
  • finalize the tracing buffers;

    The application may also spontaneously provide information to the pipe:

  • a buffer is full;
  • new tracepoints or markers are appearing or disappearing, because modules are dynamically loaded or unloaded;

    In addition, the tracing control application should be notified when the application exits (to save the content of buffers if the application is crashing). Such notification may be obtained through utrace.

    This tracing scheme plans to use a direct function call to tracing into buffers mapped in user-space. This should be an order of magnitude faster than the current Dtrace implementation (c.f. Dtrace information on the TracingWiki) which uses a breakpoint to perform both dynamic and static user-space tracing. Performance comparison of a function call vs the int3 approach is available at Markers vs int3 performance comparison (see "Conclusion"). Tracing of Java application is planned to be done through a JNI interface. Linking standard low-level C tracing library to the application within a JNI adaptation class will be required to trace Java events. This has been prototyped in the past. The work is available here for older LTTng versions :