1 The LTTng Documentation
2 =======================
3 Philippe Proulx <pproulx@efficios.com>
7 include::../common/copyright.txt[]
10 include::../common/warning-not-maintained.txt[]
13 include::../common/welcome.txt[]
16 include::../common/audience.txt[]
20 === What's in this documentation?
22 The LTTng Documentation is divided into the following sections:
24 * **<<nuts-and-bolts,Nuts and bolts>>** explains the
25 rudiments of software tracing and the rationale behind the
28 You can skip this section if you’re familiar with software tracing and
29 with the LTTng project.
31 * **<<installing-lttng,Installation>>** describes the steps to
32 install the LTTng packages on common Linux distributions and from
35 You can skip this section if you already properly installed LTTng on
38 * **<<getting-started,Quick start>>** is a concise guide to
39 getting started quickly with LTTng kernel and user space tracing.
41 We recommend this section if you're new to LTTng or to software tracing
44 You can skip this section if you're not new to LTTng.
46 * **<<core-concepts,Core concepts>>** explains the concepts at
49 It's a good idea to become familiar with the core concepts
50 before attempting to use the toolkit.
52 * **<<plumbing,Components of LTTng>>** describes the various components
53 of the LTTng machinery, like the daemons, the libraries, and the
54 command-line interface.
55 * **<<instrumenting,Instrumentation>>** shows different ways to
56 instrument user applications and the Linux kernel.
58 Instrumenting source code is essential to provide a meaningful
61 You can skip this section if you do not have a programming background.
63 * **<<controlling-tracing,Tracing control>>** is divided into topics
64 which demonstrate how to use the vast array of features that
65 LTTng{nbsp}{revision} offers.
66 * **<<reference,Reference>>** contains reference tables.
67 * **<<glossary,Glossary>>** is a specialized dictionary of terms related
68 to LTTng or to the field of software tracing.
71 include::../common/convention.txt[]
74 include::../common/acknowledgements.txt[]
78 == What's new in LTTng {revision}?
80 LTTng{nbsp}{revision} bears the name _Isseki Nicho_. The result of a
81 collaboration between http://www.dieuduciel.com/[Dieu du Ciel!] and
82 Nagano-based Shiga Kogen,
83 https://www.beeradvocate.com/beer/profile/1141/53111/[_**Isseki
84 Nicho**_] is a strong Imperial Dark Saison offering a rich roasted malt
85 flavor combined with a complex fruity finish typical of Saison yeasts.
87 New features and changes in LTTng{nbsp}{revision}:
89 * **Tracing control**:
90 ** You can attach <<java-application-context,Java application-specific
91 context fields>> to a <<channel,channel>> with the
92 man:lttng-add-context(1) command:
97 $ lttng add-context --jul --type='$app.retriever:cur_msg_id'
101 Here, `$app` is the prefix of all application-specific context fields,
102 `retriever` names a _context information retriever_ defined at the
103 application level, and `cur_msg_id` names a context field read from this
106 Both the `java.util.logging` and Apache log4j <<domain,tracing domains>>
109 ** You can use Java application-specific <<adding-context,context>>
110 fields in the <<enabling-disabling-events,filter expression>> of an
111 <<event,event rule>>:
116 $ lttng enable-event --log4j my_logger \
117 --filter='$app.retriever:cur_msg_id == 23'
121 ** New `lttng status` command which is the equivalent of +lttng list
122 __CUR__+, where +__CUR__+ is the name of the current
123 <<tracing-session,tracing session>>.
125 See man:lttng-status(1).
127 ** New `lttng metadata regenerate` command to
128 <<metadata-regenerate,regenerate the metadata file of an LTTng
129 trace>> at any moment. This command is meant to be used to resample
130 the wall time following a major
131 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
132 so that a system which boots with an incorrect wall time can be
133 traced before its wall time is NTP-corrected.
135 See man:lttng-metadata(1).
137 ** New command-line interface warnings when <<event,event records>> or
138 whole sub-buffers are
139 <<channel-overwrite-mode-vs-discard-mode,lost>>. The warning messages
140 are printed when a <<tracing-session,tracing session>> is
141 <<basic-tracing-session-control,stopped>> (man:lttng-stop(1)
144 * **User space tracing**:
145 ** Shared object base address dump in order to map <<event,event
146 records>> to original source location (file and line number).
148 If you attach the `ip` and `vpid` <<adding-context,context fields>> to a
149 user space <<channel,channel>> and if you use the
150 <<liblttng-ust-dl,path:{liblttng-ust-dl.so} helper>>, you can retrieve
151 the source location where a given event record was generated.
153 The http://diamon.org/babeltrace/[Babeltrace] trace viewer supports this
154 state dump and those context fields since version 1.4 to print the
155 source location of a given event record. http://tracecompass.org/[Trace
156 Compass] also supports this since version 2.0.
158 ** A <<java-application,Java application>> which uses
159 `java.util.logging` now adds an LTTng-UST log handler to the desired
162 The previous workflow was to initialize the LTTng-UST Java agent
163 by calling `LTTngAgent.getLTTngAgent()`. This had the effect of adding
164 an LTTng-UST log handler to the root loggers.
166 ** A <<java-application,Java application>> which uses Apache log4j now
167 adds an LTTng-UST log appender to the desired log4j loggers.
169 The previous workflow was to initialize the LTTng-UST Java agent
170 by calling `LTTngAgent.getLTTngAgent()`. This had the effect of adding
171 an LTTng-UST appender to the root loggers.
173 ** Any <<java-application,Java application>> can provide
174 <<java-application-context,dynamic context fields>> while running
175 thanks to a new API provided by the <<lttng-ust-agents,LTTng-UST Java
176 agent>>. You can require LTTng to record specific context fields in
177 event records, and you can use them in the filter expression of
178 <<event,event rules>>.
180 * **Linux kernel tracing**:
181 ** The LTTng kernel modules can now be built into a Linux kernel image,
182 that is, not as loadable modules.
185 https://github.com/lttng/lttng-modules/blob/stable-{revision}/README.md#kernel-built-in-support[`README.md`]
188 ** New instrumentation:
189 *** ARM64 architecture support.
191 *** x86 `irq_vectors`.
192 ** New <<adding-context,context fields>>:
195 *** `need_reschedule`
196 *** `migratable` (specific to RT-Preempt)
197 ** Clock source plugin support for advanced cases where a custom source
198 of time is needed to timestamp LTTng event records.
200 See https://github.com/lttng/lttng-modules/blob/stable-{revision}/lttng-clock.h[`lttng-clock.h`]
201 for an overview of the small API.
204 ** The link:/man[man pages] of the man:lttng(1) command-line tool are
205 split into one man page per command (à la Git), for example:
210 $ man lttng-enable-event
214 You can also use the `--help` option of any man:lttng(1) command to
217 The content and formatting of all the LTTng man pages has improved
224 What is LTTng? As its name suggests, the _Linux Trace Toolkit: next
225 generation_ is a modern toolkit for tracing Linux systems and
226 applications. So your first question might be:
233 As the history of software engineering progressed and led to what
234 we now take for granted--complex, numerous and
235 interdependent software applications running in parallel on
236 sophisticated operating systems like Linux--the authors of such
237 components, software developers, began feeling a natural
238 urge to have tools that would ensure the robustness and good performance
239 of their masterpieces.
241 One major achievement in this field is, inarguably, the
242 https://www.gnu.org/software/gdb/[GNU debugger (GDB)],
243 an essential tool for developers to find and fix bugs. But even the best
244 debugger won't help make your software run faster, and nowadays, faster
245 software means either more work done by the same hardware, or cheaper
246 hardware for the same work.
248 A _profiler_ is often the tool of choice to identify performance
249 bottlenecks. Profiling is suitable to identify _where_ performance is
250 lost in a given software. The profiler outputs a profile, a statistical
251 summary of observed events, which you may use to discover which
252 functions took the most time to execute. However, a profiler won't
253 report _why_ some identified functions are the bottleneck. Bottlenecks
254 might only occur when specific conditions are met, conditions that are
255 sometimes impossible to capture by a statistical profiler, or impossible
256 to reproduce with an application altered by the overhead of an
257 event-based profiler. For a thorough investigation of software
258 performance issues, a history of execution is essential, with the
259 recorded values of variables and context fields you choose, and
260 with as little influence as possible on the instrumented software. This
261 is where tracing comes in handy.
263 _Tracing_ is a technique used to understand what goes on in a running
264 software system. The software used for tracing is called a _tracer_,
265 which is conceptually similar to a tape recorder. When recording,
266 specific instrumentation points placed in the software source code
267 generate events that are saved on a giant tape: a _trace_ file. You
268 can trace user applications and the operating system at the same time,
269 opening the possibility of resolving a wide range of problems that would
270 otherwise be extremely challenging.
272 Tracing is often compared to _logging_. However, tracers and loggers are
273 two different tools, serving two different purposes. Tracers are
274 designed to record much lower-level events that occur much more
275 frequently than log messages, often in the range of thousands per
276 second, with very little execution overhead. Logging is more appropriate
277 for a very high-level analysis of less frequent events: user accesses,
278 exceptional conditions (errors and warnings, for example), database
279 transactions, instant messaging communications, and such. Simply put,
280 logging is one of the many use cases that can be satisfied with tracing.
282 The list of recorded events inside a trace file can be read manually
283 like a log file for the maximum level of detail, but it is generally
284 much more interesting to perform application-specific analyses to
285 produce reduced statistics and graphs that are useful to resolve a
286 given problem. Trace viewers and analyzers are specialized tools
289 In the end, this is what LTTng is: a powerful, open source set of
290 tools to trace the Linux kernel and user applications at the same time.
291 LTTng is composed of several components actively maintained and
292 developed by its link:/community/#where[community].
295 [[lttng-alternatives]]
296 === Alternatives to noch:{LTTng}
298 Excluding proprietary solutions, a few competing software tracers
301 * https://github.com/dtrace4linux/linux[dtrace4linux] is a port of
302 Sun Microsystems's DTrace to Linux. The cmd:dtrace tool interprets
303 user scripts and is responsible for loading code into the
304 Linux kernel for further execution and collecting the outputted data.
305 * https://en.wikipedia.org/wiki/Berkeley_Packet_Filter[eBPF] is a
306 subsystem in the Linux kernel in which a virtual machine can execute
307 programs passed from the user space to the kernel. You can attach
308 such programs to tracepoints and KProbes thanks to a system call, and
309 they can output data to the user space when executed thanks to
310 different mechanisms (pipe, VM register values, and eBPF maps, to name
312 * https://www.kernel.org/doc/Documentation/trace/ftrace.txt[ftrace]
313 is the de facto function tracer of the Linux kernel. Its user
314 interface is a set of special files in sysfs.
315 * https://perf.wiki.kernel.org/[perf] is
316 a performance analyzing tool for Linux which supports hardware
317 performance counters, tracepoints, as well as other counters and
318 types of probes. perf's controlling utility is the cmd:perf command
320 * http://linux.die.net/man/1/strace[strace]
321 is a command-line utility which records system calls made by a
322 user process, as well as signal deliveries and changes of process
323 state. strace makes use of https://en.wikipedia.org/wiki/Ptrace[ptrace]
324 to fulfill its function.
325 * http://www.sysdig.org/[sysdig], like SystemTap, uses scripts to
326 analyze Linux kernel events. You write scripts, or _chisels_ in
327 sysdig's jargon, in Lua and sysdig executes them while the system is
328 being traced or afterwards. sysdig's interface is the cmd:sysdig
329 command-line tool as well as the curses-based cmd:csysdig tool.
330 * https://sourceware.org/systemtap/[SystemTap] is a Linux kernel and
331 user space tracer which uses custom user scripts to produce plain text
332 traces. SystemTap converts the scripts to the C language, and then
333 compiles them as Linux kernel modules which are loaded to produce
334 trace data. SystemTap's primary user interface is the cmd:stap
337 The main distinctive features of LTTng is that it produces correlated
338 kernel and user space traces, as well as doing so with the lowest
339 overhead amongst other solutions. It produces trace files in the
340 http://diamon.org/ctf[CTF] format, a file format optimized
341 for the production and analyses of multi-gigabyte data.
343 LTTng is the result of more than 10 years of active open source
344 development by a community of passionate developers.
345 LTTng{nbsp}{revision} is currently available on major desktop and server
348 The main interface for tracing control is a single command-line tool
349 named cmd:lttng. The latter can create several tracing sessions, enable
350 and disable events on the fly, filter events efficiently with custom
351 user expressions, start and stop tracing, and much more. LTTng can
352 record the traces on the file system or send them over the network, and
353 keep them totally or partially. You can view the traces once tracing
354 becomes inactive or in real-time.
356 <<installing-lttng,Install LTTng now>> and
357 <<getting-started,start tracing>>!
363 include::../common/warning-no-installation.txt[]
365 **LTTng** is a set of software <<plumbing,components>> which interact to
366 <<instrumenting,instrument>> the Linux kernel and user applications, and
367 to <<controlling-tracing,control tracing>> (start and stop
368 tracing, enable and disable event rules, and the rest). Those
369 components are bundled into the following packages:
371 * **LTTng-tools**: Libraries and command-line interface to
373 * **LTTng-modules**: Linux kernel modules to instrument and
375 * **LTTng-UST**: Libraries and Java/Python packages to instrument and
376 trace user applications.
378 Most distributions mark the LTTng-modules and LTTng-UST packages as
379 optional when installing LTTng-tools (which is always required). Note
382 * You only need to install LTTng-modules if you intend to trace the
384 * You only need to install LTTng-UST if you intend to trace user
388 [[building-from-source]]
389 === Build from source
391 To build and install LTTng{nbsp}{revision} from source:
393 . Using your distribution's package manager, or from source, install
394 the following dependencies of LTTng-tools and LTTng-UST:
397 * https://sourceforge.net/projects/libuuid/[libuuid]
398 * http://directory.fsf.org/wiki/Popt[popt]
399 * http://liburcu.org/[Userspace RCU]
400 * http://www.xmlsoft.org/[libxml2]
403 . Download, build, and install the latest LTTng-modules{nbsp}{revision}:
409 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
410 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
411 cd lttng-modules-2.8.* &&
413 sudo make modules_install &&
418 . Download, build, and install the latest LTTng-UST{nbsp}{revision}:
424 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.8.tar.bz2 &&
425 tar -xf lttng-ust-latest-2.8.tar.bz2 &&
426 cd lttng-ust-2.8.* &&
436 .Java and Python application tracing
438 If you need to instrument and trace <<java-application,Java
439 applications>>, pass the `--enable-java-agent-jul`,
440 `--enable-java-agent-log4j`, or `--enable-java-agent-all` options to the
441 `configure` script, depending on which Java logging framework you use.
443 If you need to instrument and trace <<python-application,Python
444 applications>>, pass the `--enable-python-agent` option to the
445 `configure` script. You can set the `PYTHON` environment variable to the
446 path to the Python interpreter for which to install the LTTng-UST Python
454 By default, LTTng-UST libraries are installed to
455 dir:{/usr/local/lib}, which is the de facto directory in which to
456 keep self-compiled and third-party libraries.
458 When <<building-tracepoint-providers-and-user-application,linking an
459 instrumented user application with `liblttng-ust`>>:
461 * Append `/usr/local/lib` to the env:LD_LIBRARY_PATH environment
463 * Pass the `-L/usr/local/lib` and `-Wl,-rpath,/usr/local/lib` options to
464 man:gcc(1), man:g++(1), or man:clang(1).
468 . Download, build, and install the latest LTTng-tools{nbsp}{revision}:
474 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
475 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
476 cd lttng-tools-2.8.* &&
484 TIP: The https://github.com/eepp/vlttng[vlttng tool] can do all the
485 previous steps automatically for a given version of LTTng and confine
486 the installed files in a specific directory. This can be useful to test
487 LTTng without installing it on your system.
493 This is a short guide to get started quickly with LTTng kernel and user
496 Before you follow this guide, make sure to <<installing-lttng,install>>
499 This tutorial walks you through the steps to:
501 . <<tracing-the-linux-kernel,Trace the Linux kernel>>.
502 . <<tracing-your-own-user-application,Trace a user application>> written
504 . <<viewing-and-analyzing-your-traces,View and analyze the
508 [[tracing-the-linux-kernel]]
509 === Trace the Linux kernel
511 The following command lines start with the `#` prompt because you need
512 root privileges to trace the Linux kernel. You can also trace the kernel
513 as a regular user if your Unix user is a member of the
514 <<tracing-group,tracing group>>.
516 . Create a <<tracing-session,tracing session>> which writes its traces
517 to dir:{/tmp/my-kernel-trace}:
522 # lttng create my-kernel-session --output=/tmp/my-kernel-trace
526 . List the available kernel tracepoints and system calls:
531 # lttng list --kernel
532 # lttng list --kernel --syscall
536 . Create <<event,event rules>> which match the desired instrumentation
537 point names, for example the `sched_switch` and `sched_process_fork`
538 tracepoints, and the man:open(2) and man:close(2) system calls:
543 # lttng enable-event --kernel sched_switch,sched_process_fork
544 # lttng enable-event --kernel --syscall open,close
548 You can also create an event rule which matches _all_ the Linux kernel
549 tracepoints (this will generate a lot of data when tracing):
554 # lttng enable-event --kernel --all
558 . <<basic-tracing-session-control,Start tracing>>:
567 . Do some operation on your system for a few seconds. For example,
568 load a website, or list the files of a directory.
569 . <<basic-tracing-session-control,Stop tracing>> and destroy the
580 The man:lttng-destroy(1) command does not destroy the trace data; it
581 only destroys the state of the tracing session.
583 . For the sake of this example, make the recorded trace accessible to
589 sudo chown -R $(whoami) /tmp/my-kernel-trace
593 See <<viewing-and-analyzing-your-traces,View and analyze the
594 recorded events>> to view the recorded events.
597 [[tracing-your-own-user-application]]
598 === Trace a user application
600 This section steps you through a simple example to trace a
601 _Hello world_ program written in C.
603 To create the traceable user application:
605 . Create the tracepoint provider header file, which defines the
606 tracepoints and the events they can generate:
612 #undef TRACEPOINT_PROVIDER
613 #define TRACEPOINT_PROVIDER hello_world
615 #undef TRACEPOINT_INCLUDE
616 #define TRACEPOINT_INCLUDE "./hello-tp.h"
618 #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
621 #include <lttng/tracepoint.h>
631 ctf_string(my_string_field, my_string_arg)
632 ctf_integer(int, my_integer_field, my_integer_arg)
636 #endif /* _HELLO_TP_H */
638 #include <lttng/tracepoint-event.h>
642 . Create the tracepoint provider package source file:
648 #define TRACEPOINT_CREATE_PROBES
649 #define TRACEPOINT_DEFINE
651 #include "hello-tp.h"
655 . Build the tracepoint provider package:
660 $ gcc -c -I. hello-tp.c
664 . Create the _Hello World_ application source file:
671 #include "hello-tp.h"
673 int main(int argc, char *argv[])
677 puts("Hello, World!\nPress Enter to continue...");
680 * The following getchar() call is only placed here for the purpose
681 * of this demonstration, to pause the application in order for
682 * you to have time to list its tracepoints. It is not
688 * A tracepoint() call.
690 * Arguments, as defined in hello-tp.h:
692 * 1. Tracepoint provider name (required)
693 * 2. Tracepoint name (required)
694 * 3. my_integer_arg (first user-defined argument)
695 * 4. my_string_arg (second user-defined argument)
697 * Notice the tracepoint provider and tracepoint names are
698 * NOT strings: they are in fact parts of variables that the
699 * macros in hello-tp.h create.
701 tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
703 for (x = 0; x < argc; ++x) {
704 tracepoint(hello_world, my_first_tracepoint, x, argv[x]);
707 puts("Quitting now!");
708 tracepoint(hello_world, my_first_tracepoint, x * x, "x^2");
715 . Build the application:
724 . Link the application with the tracepoint provider package,
725 `liblttng-ust`, and `libdl`:
730 $ gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
734 Here's the whole build process:
737 .User space tracing tutorial's build steps.
738 image::ust-flow.png[]
740 To trace the user application:
742 . Run the application with a few arguments:
747 $ ./hello world and beyond
756 Press Enter to continue...
760 . Start an LTTng <<lttng-sessiond,session daemon>>:
765 $ lttng-sessiond --daemonize
769 Note that a session daemon might already be running, for example as
770 a service that the distribution's service manager started.
772 . List the available user space tracepoints:
777 $ lttng list --userspace
781 You see the `hello_world:my_first_tracepoint` tracepoint listed
782 under the `./hello` process.
784 . Create a <<tracing-session,tracing session>>:
789 $ lttng create my-user-space-session
793 . Create an <<event,event rule>> which matches the
794 `hello_world:my_first_tracepoint` event name:
799 $ lttng enable-event --userspace hello_world:my_first_tracepoint
803 . <<basic-tracing-session-control,Start tracing>>:
812 . Go back to the running `hello` application and press Enter. The
813 program executes all `tracepoint()` instrumentation points and exits.
814 . <<basic-tracing-session-control,Stop tracing>> and destroy the
825 The man:lttng-destroy(1) command does not destroy the trace data; it
826 only destroys the state of the tracing session.
828 By default, LTTng saves the traces in
829 +$LTTNG_HOME/lttng-traces/__name__-__date__-__time__+,
830 where +__name__+ is the tracing session name. The
831 env:LTTNG_HOME environment variable defaults to `$HOME` if not set.
833 See <<viewing-and-analyzing-your-traces,View and analyze the
834 recorded events>> to view the recorded events.
837 [[viewing-and-analyzing-your-traces]]
838 === View and analyze the recorded events
840 Once you have completed the <<tracing-the-linux-kernel,Trace the Linux
841 kernel>> and <<tracing-your-own-user-application,Trace a user
842 application>> tutorials, you can inspect the recorded events.
844 Many tools are available to read LTTng traces:
846 * **cmd:babeltrace** is a command-line utility which converts trace
847 formats; it supports the format that LTTng produces, CTF, as well as a
848 basic text output which can be ++grep++ed. The cmd:babeltrace command
849 is part of the http://diamon.org/babeltrace[Babeltrace] project.
850 * Babeltrace also includes
851 **https://www.python.org/[Python] bindings** so
852 that you can easily open and read an LTTng trace with your own script,
853 benefiting from the power of Python.
854 * http://tracecompass.org/[**Trace Compass**]
855 is a graphical user interface for viewing and analyzing any type of
856 logs or traces, including LTTng's.
857 * https://github.com/lttng/lttng-analyses[**LTTng analyses**] is a
858 project which includes many high-level analyses of LTTng kernel
859 traces, like scheduling statistics, interrupt frequency distribution,
860 top CPU usage, and more.
862 NOTE: This section assumes that the traces recorded during the previous
863 tutorials were saved to their default location, in the
864 dir:{$LTTNG_HOME/lttng-traces} directory. The env:LTTNG_HOME
865 environment variable defaults to `$HOME` if not set.
868 [[viewing-and-analyzing-your-traces-bt]]
869 ==== Use the cmd:babeltrace command-line tool
871 The simplest way to list all the recorded events of a trace is to pass
872 its path to cmd:babeltrace with no options:
876 $ babeltrace ~/lttng-traces/my-user-space-session*
879 cmd:babeltrace finds all traces recursively within the given path and
880 prints all their events, merging them in chronological order.
882 You can pipe the output of cmd:babeltrace into a tool like man:grep(1) for
887 $ babeltrace /tmp/my-kernel-trace | grep _switch
890 You can pipe the output of cmd:babeltrace into a tool like man:wc(1) to
891 count the recorded events:
895 $ babeltrace /tmp/my-kernel-trace | grep _open | wc --lines
899 [[viewing-and-analyzing-your-traces-bt-python]]
900 ==== Use the Babeltrace Python bindings
902 The <<viewing-and-analyzing-your-traces-bt,text output of cmd:babeltrace>>
903 is useful to isolate events by simple matching using man:grep(1) and
904 similar utilities. However, more elaborate filters, such as keeping only
905 event records with a field value falling within a specific range, are
906 not trivial to write using a shell. Moreover, reductions and even the
907 most basic computations involving multiple event records are virtually
908 impossible to implement.
910 Fortunately, Babeltrace ships with Python 3 bindings which makes it easy
911 to read the event records of an LTTng trace sequentially and compute the
914 The following script accepts an LTTng Linux kernel trace path as its
915 first argument and prints the short names of the top 5 running processes
916 on CPU 0 during the whole trace:
921 from collections import Counter
927 if len(sys.argv) != 2:
928 msg = 'Usage: python3 {} TRACEPATH'.format(sys.argv[0])
929 print(msg, file=sys.stderr)
932 # A trace collection contains one or more traces
933 col = babeltrace.TraceCollection()
935 # Add the trace provided by the user (LTTng traces always have
937 if col.add_trace(sys.argv[1], 'ctf') is None:
938 raise RuntimeError('Cannot add trace')
940 # This counter dict contains execution times:
942 # task command name -> total execution time (ns)
943 exec_times = Counter()
945 # This contains the last `sched_switch` timestamp
949 for event in col.events:
950 # Keep only `sched_switch` events
951 if event.name != 'sched_switch':
954 # Keep only events which happened on CPU 0
955 if event['cpu_id'] != 0:
959 cur_ts = event.timestamp
965 # Previous task command (short) name
966 prev_comm = event['prev_comm']
968 # Initialize entry in our dict if not yet done
969 if prev_comm not in exec_times:
970 exec_times[prev_comm] = 0
972 # Compute previous command execution time
973 diff = cur_ts - last_ts
975 # Update execution time of this command
976 exec_times[prev_comm] += diff
978 # Update last timestamp
982 for name, ns in exec_times.most_common(5):
984 print('{:20}{} s'.format(name, s))
989 if __name__ == '__main__':
990 sys.exit(0 if top5proc() else 1)
997 $ python3 top5proc.py /tmp/my-kernel-trace/kernel
1003 swapper/0 48.607245889 s
1004 chromium 7.192738188 s
1005 pavucontrol 0.709894415 s
1006 Compositor 0.660867933 s
1007 Xorg.bin 0.616753786 s
1010 Note that `swapper/0` is the "idle" process of CPU 0 on Linux; since we
1011 weren't using the CPU that much when tracing, its first position in the
1016 == [[understanding-lttng]]Core concepts
1018 From a user's perspective, the LTTng system is built on a few concepts,
1019 or objects, on which the <<lttng-cli,cmd:lttng command-line tool>>
1020 operates by sending commands to the <<lttng-sessiond,session daemon>>.
1021 Understanding how those objects relate to eachother is key in mastering
1024 The core concepts are:
1026 * <<tracing-session,Tracing session>>
1027 * <<domain,Tracing domain>>
1028 * <<channel,Channel and ring buffer>>
1029 * <<"event","Instrumentation point, event rule, event, and event record">>
1035 A _tracing session_ is a stateful dialogue between you and
1036 a <<lttng-sessiond,session daemon>>. You can
1037 <<creating-destroying-tracing-sessions,create a new tracing
1038 session>> with the `lttng create` command.
1040 Anything that you do when you control LTTng tracers happens within a
1041 tracing session. In particular, a tracing session:
1044 * Has its own set of trace files.
1045 * Has its own state of activity (started or stopped).
1046 * Has its own <<tracing-session-mode,mode>> (local, network streaming,
1048 * Has its own <<channel,channels>> which have their own
1049 <<event,event rules>>.
1052 .A _tracing session_ contains <<channel,channels>> that are members of <<domain,tracing domains>> and contain <<event,event rules>>.
1053 image::concepts.png[]
1055 Those attributes and objects are completely isolated between different
1058 A tracing session is analogous to a cash machine session:
1059 the operations you do on the banking system through the cash machine do
1060 not alter the data of other users of the same system. In the case of
1061 the cash machine, a session lasts as long as your bank card is inside.
1062 In the case of LTTng, a tracing session lasts from the `lttng create`
1063 command to the `lttng destroy` command.
1066 .Each Unix user has its own set of tracing sessions.
1067 image::many-sessions.png[]
1070 [[tracing-session-mode]]
1071 ==== Tracing session mode
1073 LTTng can send the generated trace data to different locations. The
1074 _tracing session mode_ dictates where to send it. The following modes
1075 are available in LTTng{nbsp}{revision}:
1078 LTTng writes the traces to the file system of the machine being traced
1081 Network streaming mode::
1082 LTTng sends the traces over the network to a
1083 <<lttng-relayd,relay daemon>> running on a remote system.
1086 LTTng does not write the traces by default. Instead, you can request
1087 LTTng to <<taking-a-snapshot,take a snapshot>>, that is, a copy of the
1088 current tracing buffers, and to write it to the target's file system
1089 or to send it over the network to a <<lttng-relayd,relay daemon>>
1090 running on a remote system.
1093 This mode is similar to the network streaming mode, but a live
1094 trace viewer can connect to the distant relay daemon to
1095 <<lttng-live,view event records as LTTng generates them>> by
1102 A _tracing domain_ is a namespace for event sources. A tracing domain
1103 has its own properties and features.
1105 There are currently five available tracing domains:
1109 * `java.util.logging` (JUL)
1113 You must specify a tracing domain when using some commands to avoid
1114 ambiguity. For example, since all the domains support named tracepoints
1115 as event sources (instrumentation points that you manually insert in the
1116 source code), you need to specify a tracing domain when
1117 <<enabling-disabling-events,creating an event rule>> because all the
1118 tracing domains could have tracepoints with the same names.
1120 Some features are reserved to specific tracing domains. Dynamic function
1121 entry and return instrumentation points, for example, are currently only
1122 supported in the Linux kernel tracing domain, but support for other
1123 tracing domains could be added in the future.
1125 You can create <<channel,channels>> in the Linux kernel and user space
1126 tracing domains. The other tracing domains have a single default
1131 === Channel and ring buffer
1133 A _channel_ is an object which is responsible for a set of ring buffers.
1134 Each ring buffer is divided into multiple sub-buffers. When an LTTng
1135 tracer emits an event, it can record it to one or more
1136 sub-buffers. The attributes of a channel determine what to do when
1137 there's no space left for a new event record because all sub-buffers
1138 are full, where to send a full sub-buffer, and other behaviours.
1140 A channel is always associated to a <<domain,tracing domain>>. The
1141 `java.util.logging` (JUL), log4j, and Python tracing domains each have
1142 a default channel which you cannot configure.
1144 A channel also owns <<event,event rules>>. When an LTTng tracer emits
1145 an event, it records it to the sub-buffers of all
1146 the enabled channels with a satisfied event rule, as long as those
1147 channels are part of active <<tracing-session,tracing sessions>>.
1150 [[channel-buffering-schemes]]
1151 ==== Per-user vs. per-process buffering schemes
1153 A channel has at least one ring buffer _per CPU_. LTTng always
1154 records an event to the ring buffer associated to the CPU on which it
1157 Two _buffering schemes_ are available when you
1158 <<enabling-disabling-channels,create a channel>> in the
1159 user space <<domain,tracing domain>>:
1161 Per-user buffering::
1162 Allocate one set of ring buffers--one per CPU--shared by all the
1163 instrumented processes of each Unix user.
1167 .Per-user buffering scheme.
1168 image::per-user-buffering.png[]
1171 Per-process buffering::
1172 Allocate one set of ring buffers--one per CPU--for each
1173 instrumented process.
1177 .Per-process buffering scheme.
1178 image::per-process-buffering.png[]
1181 The per-process buffering scheme tends to consume more memory than the
1182 per-user option because systems generally have more instrumented
1183 processes than Unix users running instrumented processes. However, the
1184 per-process buffering scheme ensures that one process having a high
1185 event throughput won't fill all the shared sub-buffers of the same
1188 The Linux kernel tracing domain has only one available buffering scheme
1189 which is to allocate a single set of ring buffers for the whole system.
1190 This scheme is similar to the per-user option, but with a single, global
1191 user "running" the kernel.
1194 [[channel-overwrite-mode-vs-discard-mode]]
1195 ==== Overwrite vs. discard event loss modes
1197 When an event occurs, LTTng records it to a specific sub-buffer (yellow
1198 arc in the following animation) of a specific channel's ring buffer.
1199 When there's no space left in a sub-buffer, the tracer marks it as
1200 consumable (red) and another, empty sub-buffer starts receiving the
1201 following event records. A <<lttng-consumerd,consumer daemon>>
1202 eventually consumes the marked sub-buffer (returns to white).
1205 [role="docsvg-channel-subbuf-anim"]
1210 In an ideal world, sub-buffers are consumed faster than they are filled,
1211 as is the case in the previous animation. In the real world,
1212 however, all sub-buffers can be full at some point, leaving no space to
1213 record the following events.
1215 By design, LTTng is a _non-blocking_ tracer: when no empty sub-buffer is
1216 available, it is acceptable to lose event records when the alternative
1217 would be to cause substantial delays in the instrumented application's
1218 execution. LTTng privileges performance over integrity; it aims at
1219 perturbing the traced system as little as possible in order to make
1220 tracing of subtle race conditions and rare interrupt cascades possible.
1222 When it comes to losing event records because no empty sub-buffer is
1223 available, the channel's _event loss mode_ determines what to do. The
1224 available event loss modes are:
1227 Drop the newest event records until a the tracer
1228 releases a sub-buffer.
1231 Clear the sub-buffer containing the oldest event records and start
1232 writing the newest event records there.
1234 This mode is sometimes called _flight recorder mode_ because it's
1236 https://en.wikipedia.org/wiki/Flight_recorder[flight recorder]:
1237 always keep a fixed amount of the latest data.
1239 Which mechanism you should choose depends on your context: prioritize
1240 the newest or the oldest event records in the ring buffer?
1242 Beware that, in overwrite mode, the tracer abandons a whole sub-buffer
1243 as soon as a there's no space left for a new event record, whereas in
1244 discard mode, the tracer only discards the event record that doesn't
1247 In discard mode, LTTng increments a count of lost event records when an
1248 event record is lost and saves this count to the trace. In overwrite
1249 mode, since LTTng 2.8, LTTng increments a count of lost sub-buffers when
1250 a sub-buffer is lost and saves this count to the trace. In this mode,
1251 the exact number of lost event records in those lost sub-buffers is not
1252 saved to the trace. Trace analyses can use the trace's saved discarded
1253 event record and sub-buffer counts to decide whether or not to perform
1254 the analyses even if trace data is known to be missing.
1256 There are a few ways to decrease your probability of losing event
1258 <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>> shows
1259 how you can fine-une the sub-buffer count and size of a channel to
1260 virtually stop losing event records, though at the cost of greater
1264 [[channel-subbuf-size-vs-subbuf-count]]
1265 ==== Sub-buffer count and size
1267 When you <<enabling-disabling-channels,create a channel>>, you can
1268 set its number of sub-buffers and their size.
1270 Note that there is noticeable CPU overhead introduced when
1271 switching sub-buffers (marking a full one as consumable and switching
1272 to an empty one for the following events to be recorded). Knowing this,
1273 the following list presents a few practical situations along with how
1274 to configure the sub-buffer count and size for them:
1276 * **High event throughput**: In general, prefer bigger sub-buffers to
1277 lower the risk of losing event records.
1279 Having bigger sub-buffers also ensures a lower
1280 <<channel-switch-timer,sub-buffer switching frequency>>.
1282 The number of sub-buffers is only meaningful if you create the channel
1283 in overwrite mode: in this case, if a sub-buffer overwrite happens, the
1284 other sub-buffers are left unaltered.
1286 * **Low event throughput**: In general, prefer smaller sub-buffers
1287 since the risk of losing event records is low.
1289 Because events occur less frequently, the sub-buffer switching frequency
1290 should remain low and thus the tracer's overhead should not be a
1293 * **Low memory system**: If your target system has a low memory
1294 limit, prefer fewer first, then smaller sub-buffers.
1296 Even if the system is limited in memory, you want to keep the
1297 sub-buffers as big as possible to avoid a high sub-buffer switching
1300 Note that LTTng uses http://diamon.org/ctf/[CTF] as its trace format,
1301 which means event data is very compact. For example, the average
1302 LTTng kernel event record weights about 32{nbsp}bytes. Thus, a
1303 sub-buffer size of 1{nbsp}MiB is considered big.
1305 The previous situations highlight the major trade-off between a few big
1306 sub-buffers and more, smaller sub-buffers: sub-buffer switching
1307 frequency vs. how much data is lost in overwrite mode. Assuming a
1308 constant event throughput and using the overwrite mode, the two
1309 following configurations have the same ring buffer total size:
1312 [role="docsvg-channel-subbuf-size-vs-count-anim"]
1317 * **2 sub-buffers of 4{nbsp}MiB each**: Expect a very low sub-buffer
1318 switching frequency, but if a sub-buffer overwrite happens, half of
1319 the event records so far (4{nbsp}MiB) are definitely lost.
1320 * **8 sub-buffers of 1{nbsp}MiB each**: Expect 4{nbsp}times the tracer's
1321 overhead as the previous configuration, but if a sub-buffer
1322 overwrite happens, only the eighth of event records so far are
1325 In discard mode, the sub-buffers count parameter is pointless: use two
1326 sub-buffers and set their size according to the requirements of your
1330 [[channel-switch-timer]]
1331 ==== Switch timer period
1333 The _switch timer period_ is an important configurable attribute of
1334 a channel to ensure periodic sub-buffer flushing.
1336 When the _switch timer_ expires, a sub-buffer switch happens. You can
1337 set the switch timer period attribute when you
1338 <<enabling-disabling-channels,create a channel>> to ensure that event
1339 data is consumed and committed to trace files or to a distant relay
1340 daemon periodically in case of a low event throughput.
1343 [role="docsvg-channel-switch-timer"]
1348 This attribute is also convenient when you use big sub-buffers to cope
1349 with a sporadic high event throughput, even if the throughput is
1353 [[channel-read-timer]]
1354 ==== Read timer period
1356 By default, the LTTng tracers use a notification mechanism to signal a
1357 full sub-buffer so that a consumer daemon can consume it. When such
1358 notifications must be avoided, for example in real-time applications,
1359 you can use the channel's _read timer_ instead. When the read timer
1360 fires, the <<lttng-consumerd,consumer daemon>> checks for full,
1361 consumable sub-buffers.
1364 [[tracefile-rotation]]
1365 ==== Trace file count and size
1367 By default, trace files can grow as large as needed. You can set the
1368 maximum size of each trace file that a channel writes when you
1369 <<enabling-disabling-channels,create a channel>>. When the size of
1370 a trace file reaches the channel's fixed maximum size, LTTng creates
1371 another file to contain the next event records. LTTng appends a file
1372 count to each trace file name in this case.
1374 If you set the trace file size attribute when you create a channel, the
1375 maximum number of trace files that LTTng creates is _unlimited_ by
1376 default. To limit them, you can also set a maximum number of trace
1377 files. When the number of trace files reaches the channel's fixed
1378 maximum count, the oldest trace file is overwritten. This mechanism is
1379 called _trace file rotation_.
1383 === Instrumentation point, event rule, event, and event record
1385 An _event rule_ is a set of conditions which must be **all** satisfied
1386 for LTTng to record an occuring event.
1388 You set the conditions when you <<enabling-disabling-events,create
1391 You always attach an event rule to <<channel,channel>> when you create
1394 When an event passes the conditions of an event rule, LTTng records it
1395 in one of the attached channel's sub-buffers.
1397 The available conditions, as of LTTng{nbsp}{revision}, are:
1399 * The event rule _is enabled_.
1400 * The instrumentation point's type _is{nbsp}T_.
1401 * The instrumentation point's name (sometimes called _event name_)
1402 _matches{nbsp}N_, but _is not{nbsp}E_.
1403 * The instrumentation point's log level _is as severe as{nbsp}L_, or
1404 _is exactly{nbsp}L_.
1405 * The fields of the event's payload _satisfy_ a filter
1406 expression{nbsp}__F__.
1408 As you can see, all the conditions but the dynamic filter are related to
1409 the event rule's status or to the instrumentation point, not to the
1410 occurring events. This is why, without a filter, checking if an event
1411 passes an event rule is not a dynamic task: when you create or modify an
1412 event rule, all the tracers of its tracing domain enable or disable the
1413 instrumentation points themselves once. This is possible because the
1414 attributes of an instrumentation point (type, name, and log level) are
1415 defined statically. In other words, without a dynamic filter, the tracer
1416 _does not evaluate_ the arguments of an instrumentation point unless it
1417 matches an enabled event rule.
1419 Note that, for LTTng to record an event, the <<channel,channel>> to
1420 which a matching event rule is attached must also be enabled, and the
1421 tracing session owning this channel must be active.
1424 .Logical path from an instrumentation point to an event record.
1425 image::event-rule.png[]
1427 .Event, event record, or event rule?
1429 With so many similar terms, it's easy to get confused.
1431 An **event** is the consequence of the execution of an _instrumentation
1432 point_, like a tracepoint that you manually place in some source code,
1433 or a Linux kernel KProbe. An event is said to _occur_ at a specific
1434 time. Different actions can be taken upon the occurrence of an event,
1435 like record the event's payload to a buffer.
1437 An **event record** is the representation of an event in a sub-buffer. A
1438 tracer is responsible for capturing the payload of an event, current
1439 context variables, the event's ID, and the event's timestamp. LTTng
1440 can append this sub-buffer to a trace file.
1442 An **event rule** is a set of conditions which must all be satisfied for
1443 LTTng to record an occuring event. Events still occur without
1444 satisfying event rules, but LTTng does not record them.
1449 == Components of noch:{LTTng}
1451 The second _T_ in _LTTng_ stands for _toolkit_: it would be wrong
1452 to call LTTng a simple _tool_ since it is composed of multiple
1453 interacting components. This section describes those components,
1454 explains their respective roles, and shows how they connect together to
1455 form the LTTng ecosystem.
1457 The following diagram shows how the most important components of LTTng
1458 interact with user applications, the Linux kernel, and you:
1461 .Control and trace data paths between LTTng components.
1462 image::plumbing.png[]
1464 The LTTng project incorporates:
1466 * **LTTng-tools**: Libraries and command-line interface to
1467 control tracing sessions.
1468 ** <<lttng-sessiond,Session daemon>> (man:lttng-sessiond(8)).
1469 ** <<lttng-consumerd,Consumer daemon>> (cmd:lttng-consumerd).
1470 ** <<lttng-relayd,Relay daemon>> (man:lttng-relayd(8)).
1471 ** <<liblttng-ctl-lttng,Tracing control library>> (`liblttng-ctl`).
1472 ** <<lttng-cli,Tracing control command-line tool>> (man:lttng(1)).
1473 * **LTTng-UST**: Libraries and Java/Python packages to trace user
1475 ** <<lttng-ust,User space tracing library>> (`liblttng-ust`) and its
1476 headers to instrument and trace any native user application.
1477 ** <<prebuilt-ust-helpers,Preloadable user space tracing helpers>>:
1478 *** `liblttng-ust-libc-wrapper`
1479 *** `liblttng-ust-pthread-wrapper`
1480 *** `liblttng-ust-cyg-profile`
1481 *** `liblttng-ust-cyg-profile-fast`
1482 *** `liblttng-ust-dl`
1483 ** User space tracepoint provider source files generator command-line
1484 tool (man:lttng-gen-tp(1)).
1485 ** <<lttng-ust-agents,LTTng-UST Java agent>> to instrument and trace
1486 Java applications using `java.util.logging` or
1487 Apache log4j 1.2 logging.
1488 ** <<lttng-ust-agents,LTTng-UST Python agent>> to instrument
1489 Python applications using the standard `logging` package.
1490 * **LTTng-modules**: <<lttng-modules,Linux kernel modules>> to trace
1492 ** LTTng kernel tracer module.
1493 ** Tracing ring buffer kernel modules.
1494 ** Probe kernel modules.
1495 ** LTTng logger kernel module.
1499 === Tracing control command-line interface
1502 .The tracing control command-line interface.
1503 image::plumbing-lttng-cli.png[]
1505 The _man:lttng(1) command-line tool_ is the standard user interface to
1506 control LTTng <<tracing-session,tracing sessions>>. The cmd:lttng tool
1507 is part of LTTng-tools.
1509 The cmd:lttng tool is linked with
1510 <<liblttng-ctl-lttng,`liblttng-ctl`>> to communicate with
1511 one or more <<lttng-sessiond,session daemons>> behind the scenes.
1513 The cmd:lttng tool has a Git-like interface:
1517 $ lttng <GENERAL OPTIONS> <COMMAND> <COMMAND OPTIONS>
1520 The <<controlling-tracing,Tracing control>> section explores the
1521 available features of LTTng using the cmd:lttng tool.
1524 [[liblttng-ctl-lttng]]
1525 === Tracing control library
1528 .The tracing control library.
1529 image::plumbing-liblttng-ctl.png[]
1531 The _LTTng control library_, `liblttng-ctl`, is used to communicate
1532 with a <<lttng-sessiond,session daemon>> using a C API that hides the
1533 underlying protocol's details. `liblttng-ctl` is part of LTTng-tools.
1535 The <<lttng-cli,cmd:lttng command-line tool>>
1536 is linked with `liblttng-ctl`.
1538 You can use `liblttng-ctl` in C or $$C++$$ source code by including its
1543 #include <lttng/lttng.h>
1546 Some objects are referenced by name (C string), such as tracing
1547 sessions, but most of them require to create a handle first using
1548 `lttng_create_handle()`.
1550 The best available developer documentation for `liblttng-ctl` is, as of
1551 LTTng{nbsp}{revision}, its installed header files. Every function and
1552 structure is thoroughly documented.
1556 === User space tracing library
1559 .The user space tracing library.
1560 image::plumbing-liblttng-ust.png[]
1562 The _user space tracing library_, `liblttng-ust` (see man:lttng-ust(3)),
1563 is the LTTng user space tracer. It receives commands from a
1564 <<lttng-sessiond,session daemon>>, for example to
1565 enable and disable specific instrumentation points, and writes event
1566 records to ring buffers shared with a
1567 <<lttng-consumerd,consumer daemon>>.
1568 `liblttng-ust` is part of LTTng-UST.
1570 Public C header files are installed beside `liblttng-ust` to
1571 instrument any <<c-application,C or $$C++$$ application>>.
1573 <<lttng-ust-agents,LTTng-UST agents>>, which are regular Java and Python
1574 packages, use their own library providing tracepoints which is
1575 linked with `liblttng-ust`.
1577 An application or library does not have to initialize `liblttng-ust`
1578 manually: its constructor does the necessary tasks to properly register
1579 to a session daemon. The initialization phase also enables the
1580 instrumentation points matching the <<event,event rules>> that you
1584 [[lttng-ust-agents]]
1585 === User space tracing agents
1588 .The user space tracing agents.
1589 image::plumbing-lttng-ust-agents.png[]
1591 The _LTTng-UST Java and Python agents_ are regular Java and Python
1592 packages which add LTTng tracing capabilities to the
1593 native logging frameworks. The LTTng-UST agents are part of LTTng-UST.
1595 In the case of Java, the
1596 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[`java.util.logging`
1597 core logging facilities] and
1598 https://logging.apache.org/log4j/1.2/[Apache log4j 1.2] are supported.
1599 Note that Apache Log4{nbsp}2 is not supported.
1601 In the case of Python, the standard
1602 https://docs.python.org/3/library/logging.html[`logging`] package
1603 is supported. Both Python 2 and Python 3 modules can import the
1604 LTTng-UST Python agent package.
1606 The applications using the LTTng-UST agents are in the
1607 `java.util.logging` (JUL),
1608 log4j, and Python <<domain,tracing domains>>.
1610 Both agents use the same mechanism to trace the log statements. When an
1611 agent is initialized, it creates a log handler that attaches to the root
1612 logger. The agent also registers to a <<lttng-sessiond,session daemon>>.
1613 When the application executes a log statement, it is passed to the
1614 agent's log handler by the root logger. The agent's log handler calls a
1615 native function in a tracepoint provider package shared library linked
1616 with <<lttng-ust,`liblttng-ust`>>, passing the formatted log message and
1617 other fields, like its logger name and its log level. This native
1618 function contains a user space instrumentation point, hence tracing the
1621 The log level condition of an
1622 <<event,event rule>> is considered when tracing
1623 a Java or a Python application, and it's compatible with the standard
1624 JUL, log4j, and Python log levels.
1628 === LTTng kernel modules
1631 .The LTTng kernel modules.
1632 image::plumbing-lttng-modules.png[]
1634 The _LTTng kernel modules_ are a set of Linux kernel modules
1635 which implement the kernel tracer of the LTTng project. The LTTng
1636 kernel modules are part of LTTng-modules.
1638 The LTTng kernel modules include:
1640 * A set of _probe_ modules.
1642 Each module attaches to a specific subsystem
1643 of the Linux kernel using its tracepoint instrument points. There are
1644 also modules to attach to the entry and return points of the Linux
1645 system call functions.
1647 * _Ring buffer_ modules.
1649 A ring buffer implementation is provided as kernel modules. The LTTng
1650 kernel tracer writes to the ring buffer; a
1651 <<lttng-consumerd,consumer daemon>> reads from the ring buffer.
1653 * The _LTTng kernel tracer_ module.
1654 * The _LTTng logger_ module.
1656 The LTTng logger module implements the special path:{/proc/lttng-logger}
1657 file so that any executable can generate LTTng events by opening and
1658 writing to this file.
1660 See <<proc-lttng-logger-abi,LTTng logger>>.
1662 Generally, you do not have to load the LTTng kernel modules manually
1663 (using man:modprobe(8), for example): a root <<lttng-sessiond,session
1664 daemon>> loads the necessary modules when starting. If you have extra
1665 probe modules, you can specify to load them to the session daemon on
1668 The LTTng kernel modules are installed in
1669 +/usr/lib/modules/__release__/extra+ by default, where +__release__+ is
1670 the kernel release (see `uname --kernel-release`).
1677 .The session daemon.
1678 image::plumbing-sessiond.png[]
1680 The _session daemon_, man:lttng-sessiond(8), is a daemon responsible for
1681 managing tracing sessions and for controlling the various components of
1682 LTTng. The session daemon is part of LTTng-tools.
1684 The session daemon sends control requests to and receives control
1687 * The <<lttng-ust,user space tracing library>>.
1689 Any instance of the user space tracing library first registers to
1690 a session daemon. Then, the session daemon can send requests to
1691 this instance, such as:
1694 ** Get the list of tracepoints.
1695 ** Share an <<event,event rule>> so that the user space tracing library
1696 can enable or disable tracepoints. Amongst the possible conditions
1697 of an event rule is a filter expression which `liblttng-ust` evalutes
1698 when an event occurs.
1699 ** Share <<channel,channel>> attributes and ring buffer locations.
1702 The session daemon and the user space tracing library use a Unix
1703 domain socket for their communication.
1705 * The <<lttng-ust-agents,user space tracing agents>>.
1707 Any instance of a user space tracing agent first registers to
1708 a session daemon. Then, the session daemon can send requests to
1709 this instance, such as:
1712 ** Get the list of loggers.
1713 ** Enable or disable a specific logger.
1716 The session daemon and the user space tracing agent use a TCP connection
1717 for their communication.
1719 * The <<lttng-modules,LTTng kernel tracer>>.
1720 * The <<lttng-consumerd,consumer daemon>>.
1722 The session daemon sends requests to the consumer daemon to instruct
1723 it where to send the trace data streams, amongst other information.
1725 * The <<lttng-relayd,relay daemon>>.
1727 The session daemon receives commands from the
1728 <<liblttng-ctl-lttng,tracing control library>>.
1730 The root session daemon loads the appropriate
1731 <<lttng-modules,LTTng kernel modules>> on startup. It also spawns
1732 a <<lttng-consumerd,consumer daemon>> as soon as you create
1733 an <<event,event rule>>.
1735 The session daemon does not send and receive trace data: this is the
1736 role of the <<lttng-consumerd,consumer daemon>> and
1737 <<lttng-relayd,relay daemon>>. It does, however, generate the
1738 http://diamon.org/ctf/[CTF] metadata stream.
1740 Each Unix user can have its own session daemon instance. The
1741 tracing sessions managed by different session daemons are completely
1744 The root user's session daemon is the only one which is
1745 allowed to control the LTTng kernel tracer, and its spawned consumer
1746 daemon is the only one which is allowed to consume trace data from the
1747 LTTng kernel tracer. Note, however, that any Unix user which is a member
1748 of the <<tracing-group,tracing group>> is allowed
1749 to create <<channel,channels>> in the
1750 Linux kernel <<domain,tracing domain>>, and thus to trace the Linux
1753 The <<lttng-cli,cmd:lttng command-line tool>> automatically starts a
1754 session daemon when using its `create` command if none is currently
1755 running. You can also start the session daemon manually.
1762 .The consumer daemon.
1763 image::plumbing-consumerd.png[]
1765 The _consumer daemon_, cmd:lttng-consumerd, is a daemon which shares
1766 ring buffers with user applications or with the LTTng kernel modules to
1767 collect trace data and send it to some location (on disk or to a
1768 <<lttng-relayd,relay daemon>> over the network). The consumer daemon
1769 is part of LTTng-tools.
1771 You do not start a consumer daemon manually: a consumer daemon is always
1772 spawned by a <<lttng-sessiond,session daemon>> as soon as you create an
1773 <<event,event rule>>, that is, before you start tracing. When you kill
1774 its owner session daemon, the consumer daemon also exits because it is
1775 the session daemon's child process. Command-line options of
1776 man:lttng-sessiond(8) target the consumer daemon process.
1778 There are up to two running consumer daemons per Unix user, whereas only
1779 one session daemon can run per user. This is because each process can be
1780 either 32-bit or 64-bit: if the target system runs a mixture of 32-bit
1781 and 64-bit processes, it is more efficient to have separate
1782 corresponding 32-bit and 64-bit consumer daemons. The root user is an
1783 exception: it can have up to _three_ running consumer daemons: 32-bit
1784 and 64-bit instances for its user applications, and one more
1785 reserved for collecting kernel trace data.
1793 image::plumbing-relayd.png[]
1795 The _relay daemon_, man:lttng-relayd(8), is a daemon acting as a bridge
1796 between remote session and consumer daemons, local trace files, and a
1797 remote live trace viewer. The relay daemon is part of LTTng-tools.
1799 The main purpose of the relay daemon is to implement a receiver of
1800 <<sending-trace-data-over-the-network,trace data over the network>>.
1801 This is useful when the target system does not have much file system
1802 space to record trace files locally.
1804 The relay daemon is also a server to which a
1805 <<lttng-live,live trace viewer>> can
1806 connect. The live trace viewer sends requests to the relay daemon to
1807 receive trace data as the target system emits events. The
1808 communication protocol is named _LTTng live_; it is used over TCP
1811 Note that you can start the relay daemon on the target system directly.
1812 This is the setup of choice when the use case is to view events as
1813 the target system emits them without the need of a remote system.
1817 == [[using-lttng]]Instrumentation
1819 There are many examples of tracing and monitoring in our everyday life:
1821 * You have access to real-time and historical weather reports and
1822 forecasts thanks to weather stations installed around the country.
1823 * You know your heart is safe thanks to an electrocardiogram.
1824 * You make sure not to drive your car too fast and to have enough fuel
1825 to reach your destination thanks to gauges visible on your dashboard.
1827 All the previous examples have something in common: they rely on
1828 **instruments**. Without the electrodes attached to the surface of your
1829 body's skin, cardiac monitoring is futile.
1831 LTTng, as a tracer, is no different from those real life examples. If
1832 you're about to trace a software system or, in other words, record its
1833 history of execution, you better have **instrumentation points** in the
1834 subject you're tracing, that is, the actual software.
1836 Various ways were developed to instrument a piece of software for LTTng
1837 tracing. The most straightforward one is to manually place
1838 instrumentation points, called _tracepoints_, in the software's source
1839 code. It is also possible to add instrumentation points dynamically in
1840 the Linux kernel <<domain,tracing domain>>.
1842 If you're only interested in tracing the Linux kernel, your
1843 instrumentation needs are probably already covered by LTTng's built-in
1844 <<lttng-modules,Linux kernel tracepoints>>. You may also wish to trace a
1845 user application which is already instrumented for LTTng tracing.
1846 In such cases, you can skip this whole section and read the topics of
1847 the <<controlling-tracing,Tracing control>> section.
1849 Many methods are available to instrument a piece of software for LTTng
1852 * <<c-application,User space instrumentation for C and $$C++$$
1854 * <<prebuilt-ust-helpers,Prebuilt user space tracing helpers>>.
1855 * <<java-application,User space Java agent>>.
1856 * <<python-application,User space Python agent>>.
1857 * <<proc-lttng-logger-abi,LTTng logger>>.
1858 * <<instrumenting-linux-kernel,LTTng kernel tracepoints>>.
1862 === [[cxx-application]]User space instrumentation for C and $$C++$$ applications
1864 The procedure to instrument a C or $$C++$$ user application with
1865 the <<lttng-ust,LTTng user space tracing library>>, `liblttng-ust`, is:
1867 . <<tracepoint-provider,Create the source files of a tracepoint provider
1869 . <<probing-the-application-source-code,Add tracepoints to
1870 the application's source code>>.
1871 . <<building-tracepoint-providers-and-user-application,Build and link
1872 a tracepoint provider package and the user application>>.
1874 If you need quick, man:printf(3)-like instrumentation, you can skip
1875 those steps and use <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>>
1878 IMPORTANT: You need to <<installing-lttng,install>> LTTng-UST to
1879 instrument a user application with `liblttng-ust`.
1882 [[tracepoint-provider]]
1883 ==== Create the source files of a tracepoint provider package
1885 A _tracepoint provider_ is a set of compiled functions which provide
1886 **tracepoints** to an application, the type of instrumentation point
1887 supported by LTTng-UST. Those functions can emit events with
1888 user-defined fields and serialize those events as event records to one
1889 or more LTTng-UST <<channel,channel>> sub-buffers. The `tracepoint()`
1890 macro, which you <<probing-the-application-source-code,insert in a user
1891 application's source code>>, calls those functions.
1893 A _tracepoint provider package_ is an object file (`.o`) or a shared
1894 library (`.so`) which contains one or more tracepoint providers.
1895 Its source files are:
1897 * One or more <<tpp-header,tracepoint provider header>> (`.h`).
1898 * A <<tpp-source,tracepoint provider package source>> (`.c`).
1900 A tracepoint provider package is dynamically linked with `liblttng-ust`,
1901 the LTTng user space tracer, at run time.
1904 .User application linked with `liblttng-ust` and containing a tracepoint provider.
1905 image::ust-app.png[]
1907 NOTE: If you need quick, man:printf(3)-like instrumentation, you can
1908 skip creating and using a tracepoint provider and use
1909 <<tracef,`tracef()`>> or <<tracelog,`tracelog()`>> instead.
1913 ===== Create a tracepoint provider header file template
1915 A _tracepoint provider header file_ contains the tracepoint
1916 definitions of a tracepoint provider.
1918 To create a tracepoint provider header file:
1920 . Start from this template:
1924 .Tracepoint provider header file template (`.h` file extension).
1926 #undef TRACEPOINT_PROVIDER
1927 #define TRACEPOINT_PROVIDER provider_name
1929 #undef TRACEPOINT_INCLUDE
1930 #define TRACEPOINT_INCLUDE "./tp.h"
1932 #if !defined(_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
1935 #include <lttng/tracepoint.h>
1938 * Use TRACEPOINT_EVENT(), TRACEPOINT_EVENT_CLASS(),
1939 * TRACEPOINT_EVENT_INSTANCE(), and TRACEPOINT_LOGLEVEL() here.
1944 #include <lttng/tracepoint-event.h>
1950 * `provider_name` with the name of your tracepoint provider.
1951 * `"tp.h"` with the name of your tracepoint provider header file.
1953 . Below the `#include <lttng/tracepoint.h>` line, put your
1954 <<defining-tracepoints,tracepoint definitions>>.
1956 Your tracepoint provider name must be unique amongst all the possible
1957 tracepoint provider names used on the same target system. We
1958 suggest to include the name of your project or company in the name,
1959 for example, `org_lttng_my_project_tpp`.
1961 TIP: [[lttng-gen-tp]]You can use the man:lttng-gen-tp(1) tool to create
1962 this boilerplate for you. When using cmd:lttng-gen-tp, all you need to
1963 write are the <<defining-tracepoints,tracepoint definitions>>.
1966 [[defining-tracepoints]]
1967 ===== Create a tracepoint definition
1969 A _tracepoint definition_ defines, for a given tracepoint:
1971 * Its **input arguments**. They are the macro parameters that the
1972 `tracepoint()` macro accepts for this particular tracepoint
1973 in the user application's source code.
1974 * Its **output event fields**. They are the sources of event fields
1975 that form the payload of any event that the execution of the
1976 `tracepoint()` macro emits for this particular tracepoint.
1978 You can create a tracepoint definition by using the
1979 `TRACEPOINT_EVENT()` macro below the `#include <lttng/tracepoint.h>`
1981 <<tpp-header,tracepoint provider header file template>>.
1983 The syntax of the `TRACEPOINT_EVENT()` macro is:
1986 .`TRACEPOINT_EVENT()` macro syntax.
1989 /* Tracepoint provider name */
1992 /* Tracepoint name */
1995 /* Input arguments */
2000 /* Output event fields */
2009 * `provider_name` with your tracepoint provider name.
2010 * `tracepoint_name` with your tracepoint name.
2011 * `arguments` with the <<tpp-def-input-args,input arguments>>.
2012 * `fields` with the <<tpp-def-output-fields,output event field>>
2015 This tracepoint emits events named `provider_name:tracepoint_name`.
2018 .Event name's length limitation
2020 The concatenation of the tracepoint provider name and the
2021 tracepoint name must not exceed **254 characters**. If it does, the
2022 instrumented application compiles and runs, but LTTng throws multiple
2023 warnings and you could experience serious issues.
2026 [[tpp-def-input-args]]The syntax of the `TP_ARGS()` macro is:
2029 .`TP_ARGS()` macro syntax.
2038 * `type` with the C type of the argument.
2039 * `arg_name` with the argument name.
2041 You can repeat `type` and `arg_name` up to 10 times to have
2042 more than one argument.
2044 .`TP_ARGS()` usage with three arguments.
2056 The `TP_ARGS()` and `TP_ARGS(void)` forms are valid to create a
2057 tracepoint definition with no input arguments.
2059 [[tpp-def-output-fields]]The `TP_FIELDS()` macro contains a list of
2060 `ctf_*()` macros. Each `ctf_*()` macro defines one event field. See
2061 man:lttng-ust(3) for a complete description of the available `ctf_*()`
2062 macros. A `ctf_*()` macro specifies the type, size, and byte order of
2065 Each `ctf_*()` macro takes an _argument expression_ parameter. This is a
2066 C expression that the tracer evalutes at the `tracepoint()` macro site
2067 in the application's source code. This expression provides a field's
2068 source of data. The argument expression can include input argument names
2069 listed in the `TP_ARGS()` macro.
2071 Each `ctf_*()` macro also takes a _field name_ parameter. Field names
2072 must be unique within a given tracepoint definition.
2074 Here's a complete tracepoint definition example:
2076 .Tracepoint definition.
2078 The following tracepoint definition defines a tracepoint which takes
2079 three input arguments and has four output event fields.
2083 #include "my-custom-structure.h"
2089 const struct my_custom_structure*, my_custom_structure,
2094 ctf_string(query_field, query)
2095 ctf_float(double, ratio_field, ratio)
2096 ctf_integer(int, recv_size, my_custom_structure->recv_size)
2097 ctf_integer(int, send_size, my_custom_structure->send_size)
2102 You can refer to this tracepoint definition with the `tracepoint()`
2103 macro in your application's source code like this:
2107 tracepoint(my_provider, my_tracepoint,
2108 my_structure, some_ratio, the_query);
2112 NOTE: The LTTng tracer only evaluates tracepoint arguments at run time
2113 if they satisfy an enabled <<event,event rule>>.
2116 [[using-tracepoint-classes]]
2117 ===== Use a tracepoint class
2119 A _tracepoint class_ is a class of tracepoints which share the same
2120 output event field definitions. A _tracepoint instance_ is one
2121 instance of such a defined tracepoint class, with its own tracepoint
2124 The <<defining-tracepoints,`TRACEPOINT_EVENT()` macro>> is actually a
2125 shorthand which defines both a tracepoint class and a tracepoint
2126 instance at the same time.
2128 When you build a tracepoint provider package, the C or $$C++$$ compiler
2129 creates one serialization function for each **tracepoint class**. A
2130 serialization function is responsible for serializing the event fields
2131 of a tracepoint to a sub-buffer when tracing.
2133 For various performance reasons, when your situation requires multiple
2134 tracepoint definitions with different names, but with the same event
2135 fields, we recommend that you manually create a tracepoint class
2136 and instantiate as many tracepoint instances as needed. One positive
2137 effect of such a design, amongst other advantages, is that all
2138 tracepoint instances of the same tracepoint class reuse the same
2139 serialization function, thus reducing
2140 https://en.wikipedia.org/wiki/Cache_pollution[cache pollution].
2142 .Use a tracepoint class and tracepoint instances.
2144 Consider the following three tracepoint definitions:
2156 ctf_integer(int, userid, userid)
2157 ctf_integer(size_t, len, len)
2169 ctf_integer(int, userid, userid)
2170 ctf_integer(size_t, len, len)
2182 ctf_integer(int, userid, userid)
2183 ctf_integer(size_t, len, len)
2188 In this case, we create three tracepoint classes, with one implicit
2189 tracepoint instance for each of them: `get_account`, `get_settings`, and
2190 `get_transaction`. However, they all share the same event field names
2191 and types. Hence three identical, yet independent serialization
2192 functions are created when you build the tracepoint provider package.
2194 A better design choice is to define a single tracepoint class and three
2195 tracepoint instances:
2199 /* The tracepoint class */
2200 TRACEPOINT_EVENT_CLASS(
2201 /* Tracepoint provider name */
2204 /* Tracepoint class name */
2207 /* Input arguments */
2213 /* Output event fields */
2215 ctf_integer(int, userid, userid)
2216 ctf_integer(size_t, len, len)
2220 /* The tracepoint instances */
2221 TRACEPOINT_EVENT_INSTANCE(
2222 /* Tracepoint provider name */
2225 /* Tracepoint class name */
2228 /* Tracepoint name */
2231 /* Input arguments */
2237 TRACEPOINT_EVENT_INSTANCE(
2246 TRACEPOINT_EVENT_INSTANCE(
2259 [[assigning-log-levels]]
2260 ===== Assign a log level to a tracepoint definition
2262 You can assign an optional _log level_ to a
2263 <<defining-tracepoints,tracepoint definition>>.
2265 Assigning different levels of severity to tracepoint definitions can
2266 be useful: when you <<enabling-disabling-events,create an event rule>>,
2267 you can target tracepoints having a log level as severe as a specific
2270 The concept of LTTng-UST log levels is similar to the levels found
2271 in typical logging frameworks:
2273 * In a logging framework, the log level is given by the function
2274 or method name you use at the log statement site: `debug()`,
2275 `info()`, `warn()`, `error()`, and so on.
2276 * In LTTng-UST, you statically assign the log level to a tracepoint
2277 definition; any `tracepoint()` macro invocation which refers to
2278 this definition has this log level.
2280 You can assign a log level to a tracepoint definition with the
2281 `TRACEPOINT_LOGLEVEL()` macro. You must use this macro _after_ the
2282 <<defining-tracepoints,`TRACEPOINT_EVENT()`>> or
2283 <<using-tracepoint-classes,`TRACEPOINT_INSTANCE()`>> macro for a given
2286 The syntax of the `TRACEPOINT_LOGLEVEL()` macro is:
2289 .`TRACEPOINT_LOGLEVEL()` macro syntax.
2291 TRACEPOINT_LOGLEVEL(provider_name, tracepoint_name, log_level)
2296 * `provider_name` with the tracepoint provider name.
2297 * `tracepoint_name` with the tracepoint name.
2298 * `log_level` with the log level to assign to the tracepoint
2299 definition named `tracepoint_name` in the `provider_name`
2300 tracepoint provider.
2302 See man:lttng-ust(3) for a list of available log level names.
2304 .Assign the `TRACE_DEBUG_UNIT` log level to a tracepoint definition.
2308 /* Tracepoint definition */
2317 ctf_integer(int, userid, userid)
2318 ctf_integer(size_t, len, len)
2322 /* Log level assignment */
2323 TRACEPOINT_LOGLEVEL(my_app, get_transaction, TRACE_DEBUG_UNIT)
2329 ===== Create a tracepoint provider package source file
2331 A _tracepoint provider package source file_ is a C source file which
2332 includes a <<tpp-header,tracepoint provider header file>> to expand its
2333 macros into event serialization and other functions.
2335 You can always use the following tracepoint provider package source
2339 .Tracepoint provider package source file template.
2341 #define TRACEPOINT_CREATE_PROBES
2346 Replace `tp.h` with the name of your <<tpp-header,tracepoint provider
2347 header file>> name. You may also include more than one tracepoint
2348 provider header file here to create a tracepoint provider package
2349 holding more than one tracepoint providers.
2352 [[probing-the-application-source-code]]
2353 ==== Add tracepoints to an application's source code
2355 Once you <<tpp-header,create a tracepoint provider header file>>, you
2356 can use the `tracepoint()` macro in your application's
2357 source code to insert the tracepoints that this header
2358 <<defining-tracepoints,defines>>.
2360 The `tracepoint()` macro takes at least two parameters: the tracepoint
2361 provider name and the tracepoint name. The corresponding tracepoint
2362 definition defines the other parameters.
2364 .`tracepoint()` usage.
2366 The following <<defining-tracepoints,tracepoint definition>> defines a
2367 tracepoint which takes two input arguments and has two output event
2371 .Tracepoint provider header file.
2373 #include "my-custom-structure.h"
2380 const char*, cmd_name
2383 ctf_string(cmd_name, cmd_name)
2384 ctf_integer(int, number_of_args, argc)
2389 You can refer to this tracepoint definition with the `tracepoint()`
2390 macro in your application's source code like this:
2393 .Application's source file.
2397 int main(int argc, char* argv[])
2399 tracepoint(my_provider, my_tracepoint, argc, argv[0]);
2405 Note how the application's source code includes
2406 the tracepoint provider header file containing the tracepoint
2407 definitions to use, path:{tp.h}.
2410 .`tracepoint()` usage with a complex tracepoint definition.
2412 Consider this complex tracepoint definition, where multiple event
2413 fields refer to the same input arguments in their argument expression
2417 .Tracepoint provider header file.
2419 /* For `struct stat` */
2420 #include <sys/types.h>
2421 #include <sys/stat.h>
2433 ctf_integer(int, my_constant_field, 23 + 17)
2434 ctf_integer(int, my_int_arg_field, my_int_arg)
2435 ctf_integer(int, my_int_arg_field2, my_int_arg * my_int_arg)
2436 ctf_integer(int, sum4_field, my_str_arg[0] + my_str_arg[1] +
2437 my_str_arg[2] + my_str_arg[3])
2438 ctf_string(my_str_arg_field, my_str_arg)
2439 ctf_integer_hex(off_t, size_field, st->st_size)
2440 ctf_float(double, size_dbl_field, (double) st->st_size)
2441 ctf_sequence_text(char, half_my_str_arg_field, my_str_arg,
2442 size_t, strlen(my_str_arg) / 2)
2447 You can refer to this tracepoint definition with the `tracepoint()`
2448 macro in your application's source code like this:
2451 .Application's source file.
2453 #define TRACEPOINT_DEFINE
2460 stat("/etc/fstab", &s);
2461 tracepoint(my_provider, my_tracepoint, 23, "Hello, World!", &s);
2467 If you look at the event record that LTTng writes when tracing this
2468 program, assuming the file size of path:{/etc/fstab} is 301{nbsp}bytes,
2469 it should look like this:
2471 .Event record fields
2473 |Field's name |Field's value
2474 |`my_constant_field` |40
2475 |`my_int_arg_field` |23
2476 |`my_int_arg_field2` |529
2478 |`my_str_arg_field` |`Hello, World!`
2479 |`size_field` |0x12d
2480 |`size_dbl_field` |301.0
2481 |`half_my_str_arg_field` |`Hello,`
2485 Sometimes, the arguments you pass to `tracepoint()` are expensive to
2486 compute--they use the call stack, for example. To avoid this
2487 computation when the tracepoint is disabled, you can use the
2488 `tracepoint_enabled()` and `do_tracepoint()` macros.
2490 The syntax of the `tracepoint_enabled()` and `do_tracepoint()` macros
2494 .`tracepoint_enabled()` and `do_tracepoint()` macros syntax.
2496 tracepoint_enabled(provider_name, tracepoint_name)
2497 do_tracepoint(provider_name, tracepoint_name, ...)
2502 * `provider_name` with the tracepoint provider name.
2503 * `tracepoint_name` with the tracepoint name.
2505 `tracepoint_enabled()` returns a non-zero value if the tracepoint named
2506 `tracepoint_name` from the provider named `provider_name` is enabled
2509 `do_tracepoint()` is like `tracepoint()`, except that it doesn't check
2510 if the tracepoint is enabled. Using `tracepoint()` with
2511 `tracepoint_enabled()` is dangerous since `tracepoint()` also contains
2512 the `tracepoint_enabled()` check, thus a race condition is
2513 possible in this situation:
2516 .Possible race condition when using `tracepoint_enabled()` with `tracepoint()`.
2518 if (tracepoint_enabled(my_provider, my_tracepoint)) {
2519 stuff = prepare_stuff();
2522 tracepoint(my_provider, my_tracepoint, stuff);
2525 If the tracepoint is enabled after the condition, then `stuff` is not
2526 prepared: the emitted event will either contain wrong data, or the whole
2527 application could crash (segmentation fault, for example).
2529 NOTE: Neither `tracepoint_enabled()` nor `do_tracepoint()` have an
2530 `STAP_PROBEV()` call. If you need it, you must emit
2534 [[building-tracepoint-providers-and-user-application]]
2535 ==== Build and link a tracepoint provider package and an application
2537 Once you have one or more <<tpp-header,tracepoint provider header
2538 files>> and a <<tpp-source,tracepoint provider package source file>>,
2539 you can create the tracepoint provider package by compiling its source
2540 file. From here, multiple build and run scenarios are possible. The
2541 following table shows common application and library configurations
2542 along with the required command lines to achieve them.
2544 In the following diagrams, we use the following file names:
2547 Executable application.
2550 Application's object file.
2553 Tracepoint provider package object file.
2556 Tracepoint provider package archive file.
2559 Tracepoint provider package shared object file.
2562 User library object file.
2565 User library shared object file.
2567 We use the following symbols in the diagrams of table below:
2570 .Symbols used in the build scenario diagrams.
2571 image::ust-sit-symbols.png[]
2573 We assume that path:{.} is part of the env:LD_LIBRARY_PATH environment
2574 variable in the following instructions.
2576 [role="growable ust-scenarios",cols="asciidoc,asciidoc"]
2577 .Common tracepoint provider package scenarios.
2579 |Scenario |Instructions
2582 The instrumented application is statically linked with
2583 the tracepoint provider package object.
2585 image::ust-sit+app-linked-with-tp-o+app-instrumented.png[]
2588 include::../common/ust-sit-step-tp-o.txt[]
2590 To build the instrumented application:
2592 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2597 #define TRACEPOINT_DEFINE
2601 . Compile the application source file:
2610 . Build the application:
2615 $ gcc -o app app.o tpp.o -llttng-ust -ldl
2619 To run the instrumented application:
2621 * Start the application:
2631 The instrumented application is statically linked with the
2632 tracepoint provider package archive file.
2634 image::ust-sit+app-linked-with-tp-a+app-instrumented.png[]
2637 To create the tracepoint provider package archive file:
2639 . Compile the <<tpp-source,tracepoint provider package source file>>:
2648 . Create the tracepoint provider package archive file:
2653 $ ar rcs tpp.a tpp.o
2657 To build the instrumented application:
2659 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2664 #define TRACEPOINT_DEFINE
2668 . Compile the application source file:
2677 . Build the application:
2682 $ gcc -o app app.o tpp.a -llttng-ust -ldl
2686 To run the instrumented application:
2688 * Start the application:
2698 The instrumented application is linked with the tracepoint provider
2699 package shared object.
2701 image::ust-sit+app-linked-with-tp-so+app-instrumented.png[]
2704 include::../common/ust-sit-step-tp-so.txt[]
2706 To build the instrumented application:
2708 . In path:{app.c}, before including path:{tpp.h}, add the following line:
2713 #define TRACEPOINT_DEFINE
2717 . Compile the application source file:
2726 . Build the application:
2731 $ gcc -o app app.o -ldl -L. -ltpp
2735 To run the instrumented application:
2737 * Start the application:
2747 The tracepoint provider package shared object is preloaded before the
2748 instrumented application starts.
2750 image::ust-sit+tp-so-preloaded+app-instrumented.png[]
2753 include::../common/ust-sit-step-tp-so.txt[]
2755 To build the instrumented application:
2757 . In path:{app.c}, before including path:{tpp.h}, add the
2763 #define TRACEPOINT_DEFINE
2764 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2768 . Compile the application source file:
2777 . Build the application:
2782 $ gcc -o app app.o -ldl
2786 To run the instrumented application with tracing support:
2788 * Preload the tracepoint provider package shared object and
2789 start the application:
2794 $ LD_PRELOAD=./libtpp.so ./app
2798 To run the instrumented application without tracing support:
2800 * Start the application:
2810 The instrumented application dynamically loads the tracepoint provider
2811 package shared object.
2813 See the <<dlclose-warning,warning about `dlclose()`>>.
2815 image::ust-sit+app-dlopens-tp-so+app-instrumented.png[]
2818 include::../common/ust-sit-step-tp-so.txt[]
2820 To build the instrumented application:
2822 . In path:{app.c}, before including path:{tpp.h}, add the
2828 #define TRACEPOINT_DEFINE
2829 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
2833 . Compile the application source file:
2842 . Build the application:
2847 $ gcc -o app app.o -ldl
2851 To run the instrumented application:
2853 * Start the application:
2863 The application is linked with the instrumented user library.
2865 The instrumented user library is statically linked with the tracepoint
2866 provider package object file.
2868 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-o+lib-instrumented.png[]
2871 include::../common/ust-sit-step-tp-o-fpic.txt[]
2873 To build the instrumented user library:
2875 . In path:{emon.c}, before including path:{tpp.h}, add the
2881 #define TRACEPOINT_DEFINE
2885 . Compile the user library source file:
2890 $ gcc -I. -fpic -c emon.c
2894 . Build the user library shared object:
2899 $ gcc -shared -o libemon.so emon.o tpp.o -llttng-ust -ldl
2903 To build the application:
2905 . Compile the application source file:
2914 . Build the application:
2919 $ gcc -o app app.o -L. -lemon
2923 To run the application:
2925 * Start the application:
2935 The application is linked with the instrumented user library.
2937 The instrumented user library is linked with the tracepoint provider
2938 package shared object.
2940 image::ust-sit+app-linked-with-lib+lib-linked-with-tp-so+lib-instrumented.png[]
2943 include::../common/ust-sit-step-tp-so.txt[]
2945 To build the instrumented user library:
2947 . In path:{emon.c}, before including path:{tpp.h}, add the
2953 #define TRACEPOINT_DEFINE
2957 . Compile the user library source file:
2962 $ gcc -I. -fpic -c emon.c
2966 . Build the user library shared object:
2971 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
2975 To build the application:
2977 . Compile the application source file:
2986 . Build the application:
2991 $ gcc -o app app.o -L. -lemon
2995 To run the application:
2997 * Start the application:
3007 The tracepoint provider package shared object is preloaded before the
3010 The application is linked with the instrumented user library.
3012 image::ust-sit+tp-so-preloaded+app-linked-with-lib+lib-instrumented.png[]
3015 include::../common/ust-sit-step-tp-so.txt[]
3017 To build the instrumented user library:
3019 . In path:{emon.c}, before including path:{tpp.h}, add the
3025 #define TRACEPOINT_DEFINE
3026 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3030 . Compile the user library source file:
3035 $ gcc -I. -fpic -c emon.c
3039 . Build the user library shared object:
3044 $ gcc -shared -o libemon.so emon.o -ldl
3048 To build the application:
3050 . Compile the application source file:
3059 . Build the application:
3064 $ gcc -o app app.o -L. -lemon
3068 To run the application with tracing support:
3070 * Preload the tracepoint provider package shared object and
3071 start the application:
3076 $ LD_PRELOAD=./libtpp.so ./app
3080 To run the application without tracing support:
3082 * Start the application:
3092 The application is linked with the instrumented user library.
3094 The instrumented user library dynamically loads the tracepoint provider
3095 package shared object.
3097 See the <<dlclose-warning,warning about `dlclose()`>>.
3099 image::ust-sit+app-linked-with-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3102 include::../common/ust-sit-step-tp-so.txt[]
3104 To build the instrumented user library:
3106 . In path:{emon.c}, before including path:{tpp.h}, add the
3112 #define TRACEPOINT_DEFINE
3113 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3117 . Compile the user library source file:
3122 $ gcc -I. -fpic -c emon.c
3126 . Build the user library shared object:
3131 $ gcc -shared -o libemon.so emon.o -ldl
3135 To build the application:
3137 . Compile the application source file:
3146 . Build the application:
3151 $ gcc -o app app.o -L. -lemon
3155 To run the application:
3157 * Start the application:
3167 The application dynamically loads the instrumented user library.
3169 The instrumented user library is linked with the tracepoint provider
3170 package shared object.
3172 See the <<dlclose-warning,warning about `dlclose()`>>.
3174 image::ust-sit+app-dlopens-lib+lib-linked-with-tp-so+lib-instrumented.png[]
3177 include::../common/ust-sit-step-tp-so.txt[]
3179 To build the instrumented user library:
3181 . In path:{emon.c}, before including path:{tpp.h}, add the
3187 #define TRACEPOINT_DEFINE
3191 . Compile the user library source file:
3196 $ gcc -I. -fpic -c emon.c
3200 . Build the user library shared object:
3205 $ gcc -shared -o libemon.so emon.o -ldl -L. -ltpp
3209 To build the application:
3211 . Compile the application source file:
3220 . Build the application:
3225 $ gcc -o app app.o -ldl -L. -lemon
3229 To run the application:
3231 * Start the application:
3241 The application dynamically loads the instrumented user library.
3243 The instrumented user library dynamically loads the tracepoint provider
3244 package shared object.
3246 See the <<dlclose-warning,warning about `dlclose()`>>.
3248 image::ust-sit+app-dlopens-lib+lib-dlopens-tp-so+lib-instrumented.png[]
3251 include::../common/ust-sit-step-tp-so.txt[]
3253 To build the instrumented user library:
3255 . In path:{emon.c}, before including path:{tpp.h}, add the
3261 #define TRACEPOINT_DEFINE
3262 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3266 . Compile the user library source file:
3271 $ gcc -I. -fpic -c emon.c
3275 . Build the user library shared object:
3280 $ gcc -shared -o libemon.so emon.o -ldl
3284 To build the application:
3286 . Compile the application source file:
3295 . Build the application:
3300 $ gcc -o app app.o -ldl -L. -lemon
3304 To run the application:
3306 * Start the application:
3316 The tracepoint provider package shared object is preloaded before the
3319 The application dynamically loads the instrumented user library.
3321 image::ust-sit+tp-so-preloaded+app-dlopens-lib+lib-instrumented.png[]
3324 include::../common/ust-sit-step-tp-so.txt[]
3326 To build the instrumented user library:
3328 . In path:{emon.c}, before including path:{tpp.h}, add the
3334 #define TRACEPOINT_DEFINE
3335 #define TRACEPOINT_PROBE_DYNAMIC_LINKAGE
3339 . Compile the user library source file:
3344 $ gcc -I. -fpic -c emon.c
3348 . Build the user library shared object:
3353 $ gcc -shared -o libemon.so emon.o -ldl
3357 To build the application:
3359 . Compile the application source file:
3368 . Build the application:
3373 $ gcc -o app app.o -L. -lemon
3377 To run the application with tracing support:
3379 * Preload the tracepoint provider package shared object and
3380 start the application:
3385 $ LD_PRELOAD=./libtpp.so ./app
3389 To run the application without tracing support:
3391 * Start the application:
3401 The application is statically linked with the tracepoint provider
3402 package object file.
3404 The application is linked with the instrumented user library.
3406 image::ust-sit+app-linked-with-tp-o+app-linked-with-lib+lib-instrumented.png[]
3409 include::../common/ust-sit-step-tp-o.txt[]
3411 To build the instrumented user library:
3413 . In path:{emon.c}, before including path:{tpp.h}, add the
3419 #define TRACEPOINT_DEFINE
3423 . Compile the user library source file:
3428 $ gcc -I. -fpic -c emon.c
3432 . Build the user library shared object:
3437 $ gcc -shared -o libemon.so emon.o
3441 To build the application:
3443 . Compile the application source file:
3452 . Build the application:
3457 $ gcc -o app app.o tpp.o -llttng-ust -ldl -L. -lemon
3461 To run the instrumented application:
3463 * Start the application:
3473 The application is statically linked with the tracepoint provider
3474 package object file.
3476 The application dynamically loads the instrumented user library.
3478 image::ust-sit+app-linked-with-tp-o+app-dlopens-lib+lib-instrumented.png[]
3481 include::../common/ust-sit-step-tp-o.txt[]
3483 To build the application:
3485 . In path:{app.c}, before including path:{tpp.h}, add the following line:
3490 #define TRACEPOINT_DEFINE
3494 . Compile the application source file:
3503 . Build the application:
3508 $ gcc -Wl,--export-dynamic -o app app.o tpp.o \
3513 The `--export-dynamic` option passed to the linker is necessary for the
3514 dynamically loaded library to ``see'' the tracepoint symbols defined in
3517 To build the instrumented user library:
3519 . Compile the user library source file:
3524 $ gcc -I. -fpic -c emon.c
3528 . Build the user library shared object:
3533 $ gcc -shared -o libemon.so emon.o
3537 To run the application:
3539 * Start the application:
3551 .Do not use man:dlclose(3) on a tracepoint provider package
3553 Never use man:dlclose(3) on any shared object which:
3555 * Is linked with, statically or dynamically, a tracepoint provider
3557 * Calls man:dlopen(3) itself to dynamically open a tracepoint provider
3558 package shared object.
3560 This is currently considered **unsafe** due to a lack of reference
3561 counting from LTTng-UST to the shared object.
3563 A known workaround (available since glibc 2.2) is to use the
3564 `RTLD_NODELETE` flag when calling man:dlopen(3) initially. This has the
3565 effect of not unloading the loaded shared object, even if man:dlclose(3)
3568 You can also preload the tracepoint provider package shared object with
3569 the env:LD_PRELOAD environment variable to overcome this limitation.
3573 [[using-lttng-ust-with-daemons]]
3574 ===== Use noch:{LTTng-UST} with daemons
3576 If your instrumented application calls man:fork(2), man:clone(2),
3577 or BSD's man:rfork(2), without a following man:exec(3)-family
3578 system call, you must preload the path:{liblttng-ust-fork.so} shared
3579 object when starting the application.
3583 $ LD_PRELOAD=liblttng-ust-fork.so ./my-app
3586 If your tracepoint provider package is
3587 a shared library which you also preload, you must put both
3588 shared objects in env:LD_PRELOAD:
3592 $ LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
3596 [[lttng-ust-pkg-config]]
3597 ===== Use noch:{pkg-config}
3599 On some distributions, LTTng-UST ships with a
3600 https://www.freedesktop.org/wiki/Software/pkg-config/[pkg-config]
3601 metadata file. If this is your case, then you can use cmd:pkg-config to
3602 build an application on the command line:
3606 $ gcc -o my-app my-app.o tp.o $(pkg-config --cflags --libs lttng-ust)
3610 [[instrumenting-32-bit-app-on-64-bit-system]]
3611 ===== [[advanced-instrumenting-techniques]]Build a 32-bit instrumented application for a 64-bit target system
3613 In order to trace a 32-bit application running on a 64-bit system,
3614 LTTng must use a dedicated 32-bit
3615 <<lttng-consumerd,consumer daemon>>.
3617 The following steps show how to build and install a 32-bit consumer
3618 daemon, which is _not_ part of the default 64-bit LTTng build, how to
3619 build and install the 32-bit LTTng-UST libraries, and how to build and
3620 link an instrumented 32-bit application in that context.
3622 To build a 32-bit instrumented application for a 64-bit target system,
3623 assuming you have a fresh target system with no installed Userspace RCU
3626 . Download, build, and install a 32-bit version of Userspace RCU:
3631 $ cd $(mktemp -d) &&
3632 wget http://lttng.org/files/urcu/userspace-rcu-latest-0.9.tar.bz2 &&
3633 tar -xf userspace-rcu-latest-0.9.tar.bz2 &&
3634 cd userspace-rcu-0.9.* &&
3635 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 &&
3637 sudo make install &&
3642 . Using your distribution's package manager, or from source, install
3643 the following 32-bit versions of the following dependencies of
3644 LTTng-tools and LTTng-UST:
3647 * https://sourceforge.net/projects/libuuid/[libuuid]
3648 * http://directory.fsf.org/wiki/Popt[popt]
3649 * http://www.xmlsoft.org/[libxml2]
3652 . Download, build, and install a 32-bit version of the latest
3653 LTTng-UST{nbsp}{revision}:
3658 $ cd $(mktemp -d) &&
3659 wget http://lttng.org/files/lttng-ust/lttng-ust-latest-2.8.tar.bz2 &&
3660 tar -xf lttng-ust-latest-2.8.tar.bz2 &&
3661 cd lttng-ust-2.8.* &&
3662 ./configure --libdir=/usr/local/lib32 \
3663 CFLAGS=-m32 CXXFLAGS=-m32 \
3664 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' &&
3666 sudo make install &&
3673 Depending on your distribution,
3674 32-bit libraries could be installed at a different location than
3675 `/usr/lib32`. For example, Debian is known to install
3676 some 32-bit libraries in `/usr/lib/i386-linux-gnu`.
3678 In this case, make sure to set `LDFLAGS` to all the
3679 relevant 32-bit library paths, for example:
3683 $ LDFLAGS='-L/usr/lib/i386-linux-gnu -L/usr/lib32'
3687 . Download the latest LTTng-tools{nbsp}{revision}, build, and install
3688 the 32-bit consumer daemon:
3693 $ cd $(mktemp -d) &&
3694 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
3695 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
3696 cd lttng-tools-2.8.* &&
3697 ./configure --libdir=/usr/local/lib32 CFLAGS=-m32 CXXFLAGS=-m32 \
3698 LDFLAGS='-L/usr/local/lib32 -L/usr/lib32' \
3699 --disable-bin-lttng --disable-bin-lttng-crash \
3700 --disable-bin-lttng-relayd --disable-bin-lttng-sessiond &&
3702 cd src/bin/lttng-consumerd &&
3703 sudo make install &&
3708 . From your distribution or from source,
3709 <<installing-lttng,install>> the 64-bit versions of
3710 LTTng-UST and Userspace RCU.
3711 . Download, build, and install the 64-bit version of the
3712 latest LTTng-tools{nbsp}{revision}:
3717 $ cd $(mktemp -d) &&
3718 wget http://lttng.org/files/lttng-tools/lttng-tools-latest-2.8.tar.bz2 &&
3719 tar -xf lttng-tools-latest-2.8.tar.bz2 &&
3720 cd lttng-tools-2.8.* &&
3721 ./configure --with-consumerd32-libdir=/usr/local/lib32 \
3722 --with-consumerd32-bin=/usr/local/lib32/lttng/libexec/lttng-consumerd &&
3724 sudo make install &&
3729 . Pass the following options to man:gcc(1), man:g++(1), or man:clang(1)
3730 when linking your 32-bit application:
3733 -m32 -L/usr/lib32 -L/usr/local/lib32 \
3734 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32
3737 For example, let's rebuild the quick start example in
3738 <<tracing-your-own-user-application,Trace a user application>> as an
3739 instrumented 32-bit application:
3744 $ gcc -m32 -c -I. hello-tp.c
3745 $ gcc -m32 -c hello.c
3746 $ gcc -m32 -o hello hello.o hello-tp.o \
3747 -L/usr/lib32 -L/usr/local/lib32 \
3748 -Wl,-rpath,/usr/lib32,-rpath,/usr/local/lib32 \
3753 No special action is required to execute the 32-bit application and
3754 to trace it: use the command-line man:lttng(1) tool as usual.
3761 man:tracef(3) is a small LTTng-UST API designed for quick,
3762 man:printf(3)-like instrumentation without the burden of
3763 <<tracepoint-provider,creating>> and
3764 <<building-tracepoint-providers-and-user-application,building>>
3765 a tracepoint provider package.
3767 To use `tracef()` in your application:
3769 . In the C or C++ source files where you need to use `tracef()`,
3770 include `<lttng/tracef.h>`:
3775 #include <lttng/tracef.h>
3779 . In the application's source code, use `tracef()` like you would use
3787 tracef("my message: %d (%s)", my_integer, my_string);
3793 . Link your application with `liblttng-ust`:
3798 $ gcc -o app app.c -llttng-ust
3802 To trace the events that `tracef()` calls emit:
3804 * <<enabling-disabling-events,Create an event rule>> which matches the
3805 `lttng_ust_tracef:*` event name:
3810 $ lttng enable-event --userspace 'lttng_ust_tracef:*'
3815 .Limitations of `tracef()`
3817 The `tracef()` utility function was developed to make user space tracing
3818 super simple, albeit with notable disadvantages compared to
3819 <<defining-tracepoints,user-defined tracepoints>>:
3821 * All the emitted events have the same tracepoint provider and
3822 tracepoint names, respectively `lttng_ust_tracef` and `event`.
3823 * There is no static type checking.
3824 * The only event record field you actually get, named `msg`, is a string
3825 potentially containing the values you passed to `tracef()`
3826 using your own format string. This also means that you cannot filter
3827 events with a custom expression at run time because there are no
3829 * Since `tracef()` uses the C standard library's man:vasprintf(3)
3830 function behind the scenes to format the strings at run time, its
3831 expected performance is lower than with user-defined tracepoints,
3832 which do not require a conversion to a string.
3834 Taking this into consideration, `tracef()` is useful for some quick
3835 prototyping and debugging, but you should not consider it for any
3836 permanent and serious applicative instrumentation.
3842 ==== Use `tracelog()`
3844 The man:tracelog(3) API is very similar to <<tracef,`tracef()`>>, with
3845 the difference that it accepts an additional log level parameter.
3847 The goal of `tracelog()` is to ease the migration from logging to
3850 To use `tracelog()` in your application:
3852 . In the C or C++ source files where you need to use `tracelog()`,
3853 include `<lttng/tracelog.h>`:
3858 #include <lttng/tracelog.h>
3862 . In the application's source code, use `tracelog()` like you would use
3863 man:printf(3), except for the first parameter which is the log
3871 tracelog(TRACE_WARNING, "my message: %d (%s)",
3872 my_integer, my_string);
3878 See man:lttng-ust(3) for a list of available log level names.
3880 . Link your application with `liblttng-ust`:
3885 $ gcc -o app app.c -llttng-ust
3889 To trace the events that `tracelog()` calls emit with a log level
3890 _as severe as_ a specific log level:
3892 * <<enabling-disabling-events,Create an event rule>> which matches the
3893 `lttng_ust_tracelog:*` event name and a minimum level
3899 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
3900 --loglevel=TRACE_WARNING
3904 To trace the events that `tracelog()` calls emit with a
3905 _specific log level_:
3907 * Create an event rule which matches the `lttng_ust_tracelog:*`
3908 event name and a specific log level:
3913 $ lttng enable-event --userspace 'lttng_ust_tracelog:*'
3914 --loglevel-only=TRACE_INFO
3919 [[prebuilt-ust-helpers]]
3920 === Prebuilt user space tracing helpers
3922 The LTTng-UST package provides a few helpers in the form of preloadable
3923 shared objects which automatically instrument system functions and
3926 The helper shared objects are normally found in dir:{/usr/lib}. If you
3927 built LTTng-UST <<building-from-source,from source>>, they are probably
3928 located in dir:{/usr/local/lib}.
3930 The installed user space tracing helpers in LTTng-UST{nbsp}{revision}
3933 path:{liblttng-ust-libc-wrapper.so}::
3934 path:{liblttng-ust-pthread-wrapper.so}::
3935 <<liblttng-ust-libc-pthread-wrapper,C{nbsp}standard library
3936 memory and POSIX threads function tracing>>.
3938 path:{liblttng-ust-cyg-profile.so}::
3939 path:{liblttng-ust-cyg-profile-fast.so}::
3940 <<liblttng-ust-cyg-profile,Function entry and exit tracing>>.
3942 path:{liblttng-ust-dl.so}::
3943 <<liblttng-ust-dl,Dynamic linker tracing>>.
3945 To use a user space tracing helper with any user application:
3947 * Preload the helper shared object when you start the application:
3952 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so my-app
3956 You can preload more than one helper:
3961 $ LD_PRELOAD=liblttng-ust-libc-wrapper.so:liblttng-ust-dl.so my-app
3967 [[liblttng-ust-libc-pthread-wrapper]]
3968 ==== Instrument C standard library memory and POSIX threads functions
3970 The path:{liblttng-ust-libc-wrapper.so} and
3971 path:{liblttng-ust-pthread-wrapper.so} helpers
3972 add instrumentation to some C standard library and POSIX
3976 .Functions instrumented by preloading path:{liblttng-ust-libc-wrapper.so}.
3978 |TP provider name |TP name |Instrumented function
3980 .6+|`lttng_ust_libc` |`malloc` |man:malloc(3)
3981 |`calloc` |man:calloc(3)
3982 |`realloc` |man:realloc(3)
3983 |`free` |man:free(3)
3984 |`memalign` |man:memalign(3)
3985 |`posix_memalign` |man:posix_memalign(3)
3989 .Functions instrumented by preloading path:{liblttng-ust-pthread-wrapper.so}.
3991 |TP provider name |TP name |Instrumented function
3993 .4+|`lttng_ust_pthread` |`pthread_mutex_lock_req` |man:pthread_mutex_lock(3p) (request time)
3994 |`pthread_mutex_lock_acq` |man:pthread_mutex_lock(3p) (acquire time)
3995 |`pthread_mutex_trylock` |man:pthread_mutex_trylock(3p)
3996 |`pthread_mutex_unlock` |man:pthread_mutex_unlock(3p)
3999 When you preload the shared object, it replaces the functions listed
4000 in the previous tables by wrappers which contain tracepoints and call
4001 the replaced functions.
4004 [[liblttng-ust-cyg-profile]]
4005 ==== Instrument function entry and exit
4007 The path:{liblttng-ust-cyg-profile*.so} helpers can add instrumentation
4008 to the entry and exit points of functions.
4010 man:gcc(1) and man:clang(1) have an option named
4011 https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html[`-finstrument-functions`]
4012 which generates instrumentation calls for entry and exit to functions.
4013 The LTTng-UST function tracing helpers,
4014 path:{liblttng-ust-cyg-profile.so} and
4015 path:{liblttng-ust-cyg-profile-fast.so}, take advantage of this feature
4016 to add tracepoints to the two generated functions (which contain
4017 `cyg_profile` in their names, hence the helper's name).
4019 To use the LTTng-UST function tracing helper, the source files to
4020 instrument must be built using the `-finstrument-functions` compiler
4023 There are two versions of the LTTng-UST function tracing helper:
4025 * **path:{liblttng-ust-cyg-profile-fast.so}** is a lightweight variant
4026 that you should only use when it can be _guaranteed_ that the
4027 complete event stream is recorded without any lost event record.
4028 Any kind of duplicate information is left out.
4030 Assuming no event record is lost, having only the function addresses on
4031 entry is enough to create a call graph, since an event record always
4032 contains the ID of the CPU that generated it.
4034 You can use a tool like man:addr2line(1) to convert function addresses
4035 back to source file names and line numbers.
4037 * **path:{liblttng-ust-cyg-profile.so}** is a more robust variant
4038 which also works in use cases where event records might get discarded or
4039 not recorded from application startup.
4040 In these cases, the trace analyzer needs more information to be
4041 able to reconstruct the program flow.
4043 See man:lttng-ust-cyg-profile(3) to learn more about the instrumentation
4044 points of this helper.
4046 All the tracepoints that this helper provides have the
4047 log level `TRACE_DEBUG_FUNCTION` (see man:lttng-ust(3)).
4049 TIP: It's sometimes a good idea to limit the number of source files that
4050 you compile with the `-finstrument-functions` option to prevent LTTng
4051 from writing an excessive amount of trace data at run time. When using
4052 man:gcc(1), you can use the
4053 `-finstrument-functions-exclude-function-list` option to avoid
4054 instrument entries and exits of specific function names.
4059 ==== Instrument the dynamic linker
4061 The path:{liblttng-ust-dl.so} helper adds instrumentation to the
4062 man:dlopen(3) and man:dlclose(3) function calls.
4064 See man:lttng-ust-dl(3) to learn more about the instrumentation points
4069 [[java-application]]
4070 === User space Java agent
4072 You can instrument any Java application which uses one of the following
4075 * The https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[**`java.util.logging`**]
4076 (JUL) core logging facilities.
4077 * http://logging.apache.org/log4j/1.2/[**Apache log4j 1.2**], since
4078 LTTng 2.6. Note that Apache Log4j{nbsp}2 is not supported.
4081 .LTTng-UST Java agent imported by a Java application.
4082 image::java-app.png[]
4084 Note that the methods described below are new in LTTng{nbsp}2.8.
4085 Previous LTTng versions use another technique.
4087 NOTE: We use http://openjdk.java.net/[OpenJDK]{nbsp}8 for development
4088 and https://ci.lttng.org/[continuous integration], thus this version is
4089 directly supported. However, the LTTng-UST Java agent is also tested
4090 with OpenJDK{nbsp}7.
4095 ==== Use the LTTng-UST Java agent for `java.util.logging`
4097 To use the LTTng-UST Java agent in a Java application which uses
4098 `java.util.logging` (JUL):
4100 . In the Java application's source code, import the LTTng-UST
4101 log handler package for `java.util.logging`:
4106 import org.lttng.ust.agent.jul.LttngLogHandler;
4110 . Create an LTTng-UST JUL log handler:
4115 Handler lttngUstLogHandler = new LttngLogHandler();
4119 . Add this handler to the JUL loggers which should emit LTTng events:
4124 Logger myLogger = Logger.getLogger("some-logger");
4126 myLogger.addHandler(lttngUstLogHandler);
4130 . Use `java.util.logging` log statements and configuration as usual.
4131 The loggers with an attached LTTng-UST log handler can emit
4134 . Before exiting the application, remove the LTTng-UST log handler from
4135 the loggers attached to it and call its `close()` method:
4140 myLogger.removeHandler(lttngUstLogHandler);
4141 lttngUstLogHandler.close();
4145 This is not strictly necessary, but it is recommended for a clean
4146 disposal of the handler's resources.
4148 . Include the LTTng-UST Java agent's common and JUL-specific JAR files,
4149 path:{lttng-ust-agent-common.jar} and path:{lttng-ust-agent-jul.jar},
4151 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4152 path] when you build the Java application.
4154 The JAR files are typically located in dir:{/usr/share/java}.
4156 IMPORTANT: The LTTng-UST Java agent must be
4157 <<installing-lttng,installed>> for the logging framework your
4160 .Use the LTTng-UST Java agent for `java.util.logging`.
4165 import java.io.IOException;
4166 import java.util.logging.Handler;
4167 import java.util.logging.Logger;
4168 import org.lttng.ust.agent.jul.LttngLogHandler;
4172 private static final int answer = 42;
4174 public static void main(String[] argv) throws Exception
4177 Logger logger = Logger.getLogger("jello");
4179 // Create an LTTng-UST log handler
4180 Handler lttngUstLogHandler = new LttngLogHandler();
4182 // Add the LTTng-UST log handler to our logger
4183 logger.addHandler(lttngUstLogHandler);
4186 logger.info("some info");
4187 logger.warning("some warning");
4189 logger.finer("finer information; the answer is " + answer);
4191 logger.severe("error!");
4193 // Not mandatory, but cleaner
4194 logger.removeHandler(lttngUstLogHandler);
4195 lttngUstLogHandler.close();
4204 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4207 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4208 <<enabling-disabling-events,create an event rule>> matching the
4209 `jello` JUL logger, and <<basic-tracing-session-control,start tracing>>:
4214 $ lttng enable-event --jul jello
4218 Run the compiled class:
4222 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4225 <<basic-tracing-session-control,Stop tracing>> and inspect the
4235 In the resulting trace, an <<event,event record>> generated by a Java
4236 application using `java.util.logging` is named `lttng_jul:event` and
4237 has the following fields:
4240 Log record's message.
4246 Name of the class in which the log statement was executed.
4249 Name of the method in which the log statement was executed.
4252 Logging time (timestamp in milliseconds).
4255 Log level integer value.
4258 ID of the thread in which the log statement was executed.
4260 You can use the opt:lttng-enable-event(1):--loglevel or
4261 opt:lttng-enable-event(1):--loglevel-only option of the
4262 man:lttng-enable-event(1) command to target a range of JUL log levels
4263 or a specific JUL log level.
4268 ==== Use the LTTng-UST Java agent for Apache log4j
4270 To use the LTTng-UST Java agent in a Java application which uses
4273 . In the Java application's source code, import the LTTng-UST
4274 log appender package for Apache log4j:
4279 import org.lttng.ust.agent.log4j.LttngLogAppender;
4283 . Create an LTTng-UST log4j log appender:
4288 Appender lttngUstLogAppender = new LttngLogAppender();
4292 . Add this appender to the log4j loggers which should emit LTTng events:
4297 Logger myLogger = Logger.getLogger("some-logger");
4299 myLogger.addAppender(lttngUstLogAppender);
4303 . Use Apache log4j log statements and configuration as usual. The
4304 loggers with an attached LTTng-UST log appender can emit LTTng events.
4306 . Before exiting the application, remove the LTTng-UST log appender from
4307 the loggers attached to it and call its `close()` method:
4312 myLogger.removeAppender(lttngUstLogAppender);
4313 lttngUstLogAppender.close();
4317 This is not strictly necessary, but it is recommended for a clean
4318 disposal of the appender's resources.
4320 . Include the LTTng-UST Java agent's common and log4j-specific JAR
4321 files, path:{lttng-ust-agent-common.jar} and
4322 path:{lttng-ust-agent-log4j.jar}, in the
4323 https://docs.oracle.com/javase/tutorial/essential/environment/paths.html[class
4324 path] when you build the Java application.
4326 The JAR files are typically located in dir:{/usr/share/java}.
4328 IMPORTANT: The LTTng-UST Java agent must be
4329 <<installing-lttng,installed>> for the logging framework your
4332 .Use the LTTng-UST Java agent for Apache log4j.
4337 import org.apache.log4j.Appender;
4338 import org.apache.log4j.Logger;
4339 import org.lttng.ust.agent.log4j.LttngLogAppender;
4343 private static final int answer = 42;
4345 public static void main(String[] argv) throws Exception
4348 Logger logger = Logger.getLogger("jello");
4350 // Create an LTTng-UST log appender
4351 Appender lttngUstLogAppender = new LttngLogAppender();
4353 // Add the LTTng-UST log appender to our logger
4354 logger.addAppender(lttngUstLogAppender);
4357 logger.info("some info");
4358 logger.warn("some warning");
4360 logger.debug("debug information; the answer is " + answer);
4362 logger.fatal("error!");
4364 // Not mandatory, but cleaner
4365 logger.removeAppender(lttngUstLogAppender);
4366 lttngUstLogAppender.close();
4372 Build this example (`$LOG4JPATH` is the path to the Apache log4j JAR
4377 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH Test.java
4380 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4381 <<enabling-disabling-events,create an event rule>> matching the
4382 `jello` log4j logger, and <<basic-tracing-session-control,start tracing>>:
4387 $ lttng enable-event --log4j jello
4391 Run the compiled class:
4395 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-log4j.jar:$LOG4JPATH:. Test
4398 <<basic-tracing-session-control,Stop tracing>> and inspect the
4408 In the resulting trace, an <<event,event record>> generated by a Java
4409 application using log4j is named `lttng_log4j:event` and
4410 has the following fields:
4413 Log record's message.
4419 Name of the class in which the log statement was executed.
4422 Name of the method in which the log statement was executed.
4425 Name of the file in which the executed log statement is located.
4428 Line number at which the log statement was executed.
4434 Log level integer value.
4437 Name of the Java thread in which the log statement was executed.
4439 You can use the opt:lttng-enable-event(1):--loglevel or
4440 opt:lttng-enable-event(1):--loglevel-only option of the
4441 man:lttng-enable-event(1) command to target a range of Apache log4j log levels
4442 or a specific log4j log level.
4446 [[java-application-context]]
4447 ==== Provide application-specific context fields in a Java application
4449 A Java application-specific context field is a piece of state provided
4450 by the application which <<adding-context,you can add>>, using the
4451 man:lttng-add-context(1) command, to each <<event,event record>>
4452 produced by the log statements of this application.
4454 For example, a given object might have a current request ID variable.
4455 You can create a context information retriever for this object and
4456 assign a name to this current request ID. You can then, using the
4457 man:lttng-add-context(1) command, add this context field by name to
4458 the JUL or log4j <<channel,channel>>.
4460 To provide application-specific context fields in a Java application:
4462 . In the Java application's source code, import the LTTng-UST
4463 Java agent context classes and interfaces:
4468 import org.lttng.ust.agent.context.ContextInfoManager;
4469 import org.lttng.ust.agent.context.IContextInfoRetriever;
4473 . Create a context information retriever class, that is, a class which
4474 implements the `IContextInfoRetriever` interface:
4479 class MyContextInfoRetriever implements IContextInfoRetriever
4482 public Object retrieveContextInfo(String key)
4484 if (key.equals("intCtx")) {
4486 } else if (key.equals("strContext")) {
4487 return "context value!";
4496 This `retrieveContextInfo()` method is the only member of the
4497 `IContextInfoRetriever` interface. Its role is to return the current
4498 value of a state by name to create a context field. The names of the
4499 context fields and which state variables they return depends on your
4502 All primitive types and objects are supported as context fields.
4503 When `retrieveContextInfo()` returns an object, the context field
4504 serializer calls its `toString()` method to add a string field to
4505 event records. The method can also return `null`, which means that
4506 no context field is available for the required name.
4508 . Register an instance of your context information retriever class to
4509 the context information manager singleton:
4514 IContextInfoRetriever cir = new MyContextInfoRetriever();
4515 ContextInfoManager cim = ContextInfoManager.getInstance();
4516 cim.registerContextInfoRetriever("retrieverName", cir);
4520 . Before exiting the application, remove your context information
4521 retriever from the context information manager singleton:
4526 ContextInfoManager cim = ContextInfoManager.getInstance();
4527 cim.unregisterContextInfoRetriever("retrieverName");
4531 This is not strictly necessary, but it is recommended for a clean
4532 disposal of some manager's resources.
4534 . Build your Java application with LTTng-UST Java agent support as
4535 usual, following the procedure for either the <<jul,JUL>> or
4536 <<log4j,Apache log4j>> framework.
4539 .Provide application-specific context fields in a Java application.
4544 import java.util.logging.Handler;
4545 import java.util.logging.Logger;
4546 import org.lttng.ust.agent.jul.LttngLogHandler;
4547 import org.lttng.ust.agent.context.ContextInfoManager;
4548 import org.lttng.ust.agent.context.IContextInfoRetriever;
4552 // Our context information retriever class
4553 private static class MyContextInfoRetriever
4554 implements IContextInfoRetriever
4557 public Object retrieveContextInfo(String key) {
4558 if (key.equals("intCtx")) {
4560 } else if (key.equals("strContext")) {
4561 return "context value!";
4568 private static final int answer = 42;
4570 public static void main(String args[]) throws Exception
4572 // Get the context information manager instance
4573 ContextInfoManager cim = ContextInfoManager.getInstance();
4575 // Create and register our context information retriever
4576 IContextInfoRetriever cir = new MyContextInfoRetriever();
4577 cim.registerContextInfoRetriever("myRetriever", cir);
4580 Logger logger = Logger.getLogger("jello");
4582 // Create an LTTng-UST log handler
4583 Handler lttngUstLogHandler = new LttngLogHandler();
4585 // Add the LTTng-UST log handler to our logger
4586 logger.addHandler(lttngUstLogHandler);
4589 logger.info("some info");
4590 logger.warning("some warning");
4592 logger.finer("finer information; the answer is " + answer);
4594 logger.severe("error!");
4596 // Not mandatory, but cleaner
4597 logger.removeHandler(lttngUstLogHandler);
4598 lttngUstLogHandler.close();
4599 cim.unregisterContextInfoRetriever("myRetriever");
4608 $ javac -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar Test.java
4611 <<creating-destroying-tracing-sessions,Create a tracing session>>
4612 and <<enabling-disabling-events,create an event rule>> matching the
4618 $ lttng enable-event --jul jello
4621 <<adding-context,Add the application-specific context fields>> to the
4626 $ lttng add-context --jul --type='$app.myRetriever:intCtx'
4627 $ lttng add-context --jul --type='$app.myRetriever:strContext'
4630 <<basic-tracing-session-control,Start tracing>>:
4637 Run the compiled class:
4641 $ java -cp /usr/share/java/jarpath/lttng-ust-agent-common.jar:/usr/share/java/jarpath/lttng-ust-agent-jul.jar:. Test
4644 <<basic-tracing-session-control,Stop tracing>> and inspect the
4656 [[python-application]]
4657 === User space Python agent
4659 You can instrument a Python 2 or Python 3 application which uses the
4660 standard https://docs.python.org/3/library/logging.html[`logging`]
4663 Each log statement emits an LTTng event once the
4664 application module imports the
4665 <<lttng-ust-agents,LTTng-UST Python agent>> package.
4668 .A Python application importing the LTTng-UST Python agent.
4669 image::python-app.png[]
4671 To use the LTTng-UST Python agent:
4673 . In the Python application's source code, import the LTTng-UST Python
4683 The LTTng-UST Python agent automatically adds its logging handler to the
4684 root logger at import time.
4686 Any log statement that the application executes before this import does
4687 not emit an LTTng event.
4689 IMPORTANT: The LTTng-UST Python agent must be
4690 <<installing-lttng,installed>>.
4692 . Use log statements and logging configuration as usual.
4693 Since the LTTng-UST Python agent adds a handler to the _root_
4694 logger, you can trace any log statement from any logger.
4696 .Use the LTTng-UST Python agent.
4707 logging.basicConfig()
4708 logger = logging.getLogger('my-logger')
4711 logger.debug('debug message')
4712 logger.info('info message')
4713 logger.warn('warn message')
4714 logger.error('error message')
4715 logger.critical('critical message')
4719 if __name__ == '__main__':
4723 NOTE: `logging.basicConfig()`, which adds to the root logger a basic
4724 logging handler which prints to the standard error stream, is not
4725 strictly required for LTTng-UST tracing to work, but in versions of
4726 Python preceding 3.2, you could see a warning message which indicates
4727 that no handler exists for the logger `my-logger`.
4729 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4730 <<enabling-disabling-events,create an event rule>> matching the
4731 `my-logger` Python logger, and <<basic-tracing-session-control,start
4737 $ lttng enable-event --python my-logger
4741 Run the Python script:
4748 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
4758 In the resulting trace, an <<event,event record>> generated by a Python
4759 application is named `lttng_python:event` and has the following fields:
4762 Logging time (string).
4765 Log record's message.
4771 Name of the function in which the log statement was executed.
4774 Line number at which the log statement was executed.
4777 Log level integer value.
4780 ID of the Python thread in which the log statement was executed.
4783 Name of the Python thread in which the log statement was executed.
4785 You can use the opt:lttng-enable-event(1):--loglevel or
4786 opt:lttng-enable-event(1):--loglevel-only option of the
4787 man:lttng-enable-event(1) command to target a range of Python log levels
4788 or a specific Python log level.
4790 When an application imports the LTTng-UST Python agent, the agent tries
4791 to register to a <<lttng-sessiond,session daemon>>. Note that you must
4792 <<start-sessiond,start the session daemon>> _before_ you run the Python
4793 application. If a session daemon is found, the agent tries to register
4794 to it during 5{nbsp}seconds, after which the application continues
4795 without LTTng tracing support. You can override this timeout value with
4796 the env:LTTNG_UST_PYTHON_REGISTER_TIMEOUT environment variable
4799 If the session daemon stops while a Python application with an imported
4800 LTTng-UST Python agent runs, the agent retries to connect and to
4801 register to a session daemon every 3{nbsp}seconds. You can override this
4802 delay with the env:LTTNG_UST_PYTHON_REGISTER_RETRY_DELAY environment
4807 [[proc-lttng-logger-abi]]
4810 The `lttng-tracer` Linux kernel module, part of
4811 <<lttng-modules,LTTng-modules>>, creates the special LTTng logger file
4812 path:{/proc/lttng-logger} when it's loaded. Any application can write
4813 text data to this file to emit an LTTng event.
4816 .An application writes to the LTTng logger file to emit an LTTng event.
4817 image::lttng-logger.png[]
4819 The LTTng logger is the quickest method--not the most efficient,
4820 however--to add instrumentation to an application. It is designed
4821 mostly to instrument shell scripts:
4825 $ echo "Some message, some $variable" > /proc/lttng-logger
4828 Any event that the LTTng logger emits is named `lttng_logger` and
4829 belongs to the Linux kernel <<domain,tracing domain>>. However, unlike
4830 other instrumentation points in the kernel tracing domain, **any Unix
4831 user** can <<enabling-disabling-events,create an event rule>> which
4832 matches its event name, not only the root user or users in the
4833 <<tracing-group,tracing group>>.
4835 To use the LTTng logger:
4837 * From any application, write text data to the path:{/proc/lttng-logger}
4840 The `msg` field of `lttng_logger` event records contains the
4843 NOTE: The maximum message length of an LTTng logger event is
4844 1024{nbsp}bytes. Writing more than this makes the LTTng logger emit more
4845 than one event to contain the remaining data.
4847 You should not use the LTTng logger to trace a user application which
4848 can be instrumented in a more efficient way, namely:
4850 * <<c-application,C and $$C++$$ applications>>.
4851 * <<java-application,Java applications>>.
4852 * <<python-application,Python applications>>.
4854 .Use the LTTng logger.
4859 echo 'Hello, World!' > /proc/lttng-logger
4861 df --human-readable --print-type / > /proc/lttng-logger
4864 <<creating-destroying-tracing-sessions,Create a tracing session>>,
4865 <<enabling-disabling-events,create an event rule>> matching the
4866 `lttng_logger` Linux kernel tracepoint, and
4867 <<basic-tracing-session-control,start tracing>>:
4872 $ lttng enable-event --kernel lttng_logger
4876 Run the Bash script:
4883 <<basic-tracing-session-control,Stop tracing>> and inspect the recorded
4894 [[instrumenting-linux-kernel]]
4895 === LTTng kernel tracepoints
4897 NOTE: This section shows how to _add_ instrumentation points to the
4898 Linux kernel. The kernel's subsystems are already thoroughly
4899 instrumented at strategic places for LTTng when you
4900 <<installing-lttng,install>> the <<lttng-modules,LTTng-modules>>
4904 There are two methods to instrument the Linux kernel:
4906 . <<linux-add-lttng-layer,Add an LTTng layer>> over an existing ftrace
4907 tracepoint which uses the `TRACE_EVENT()` API.
4909 Choose this if you want to instrumentation a Linux kernel tree with an
4910 instrumentation point compatible with ftrace, perf, and SystemTap.
4912 . Use an <<linux-lttng-tracepoint-event,LTTng-only approach>> to
4913 instrument an out-of-tree kernel module.
4915 Choose this if you don't need ftrace, perf, or SystemTap support.
4919 [[linux-add-lttng-layer]]
4920 ==== [[instrumenting-linux-kernel-itself]][[mainline-trace-event]][[lttng-adaptation-layer]]Add an LTTng layer to an existing ftrace tracepoint
4922 This section shows how to add an LTTng layer to existing ftrace
4923 instrumentation using the `TRACE_EVENT()` API.
4925 This section does not document the `TRACE_EVENT()` macro. You can
4926 read the following articles to learn more about this API:
4928 * http://lwn.net/Articles/379903/[Using the TRACE_EVENT() macro (Part 1)]
4929 * http://lwn.net/Articles/381064/[Using the TRACE_EVENT() macro (Part 2)]
4930 * http://lwn.net/Articles/383362/[Using the TRACE_EVENT() macro (Part 3)]
4932 The following procedure assumes that your ftrace tracepoints are
4933 correctly defined in their own header and that they are created in
4934 one source file using the `CREATE_TRACE_POINTS` definition.
4936 To add an LTTng layer over an existing ftrace tracepoint:
4938 . Make sure the following kernel configuration options are
4944 * `CONFIG_HIGH_RES_TIMERS`
4945 * `CONFIG_TRACEPOINTS`
4948 . Build the Linux source tree with your custom ftrace tracepoints.
4949 . Boot the resulting Linux image on your target system.
4951 Confirm that the tracepoints exist by looking for their names in the
4952 dir:{/sys/kernel/debug/tracing/events/subsys} directory, where `subsys`
4953 is your subsystem's name.
4955 . Get a copy of the latest LTTng-modules{nbsp}{revision}:
4960 $ cd $(mktemp -d) &&
4961 wget http://lttng.org/files/lttng-modules/lttng-modules-latest-2.8.tar.bz2 &&
4962 tar -xf lttng-modules-latest-2.8.tar.bz2 &&
4963 cd lttng-modules-2.8.*
4967 . In dir:{instrumentation/events/lttng-module}, relative to the root
4968 of the LTTng-modules source tree, create a header file named
4969 +__subsys__.h+ for your custom subsystem +__subsys__+ and write your
4970 LTTng-modules tracepoint definitions using the LTTng-modules
4973 Start with this template:
4977 .path:{instrumentation/events/lttng-module/my_subsys.h}
4980 #define TRACE_SYSTEM my_subsys
4982 #if !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ)
4983 #define _LTTNG_MY_SUBSYS_H
4985 #include "../../../probes/lttng-tracepoint-event.h"
4986 #include <linux/tracepoint.h>
4988 LTTNG_TRACEPOINT_EVENT(
4990 * Format is identical to TRACE_EVENT()'s version for the three
4991 * following macro parameters:
4994 TP_PROTO(int my_int, const char *my_string),
4995 TP_ARGS(my_int, my_string),
4997 /* LTTng-modules specific macros */
4999 ctf_integer(int, my_int_field, my_int)
5000 ctf_string(my_bar_field, my_bar)
5004 #endif /* !defined(_LTTNG_MY_SUBSYS_H) || defined(TRACE_HEADER_MULTI_READ) */
5006 #include "../../../probes/define_trace.h"
5010 The entries in the `TP_FIELDS()` section are the list of fields for the
5011 LTTng tracepoint. This is similar to the `TP_STRUCT__entry()` part of
5012 ftrace's `TRACE_EVENT()` macro.
5014 See <<lttng-modules-tp-fields,Tracepoint fields macros>> for a
5015 complete description of the available `ctf_*()` macros.
5017 . Create the LTTng-modules probe's kernel module C source file,
5018 +probes/lttng-probe-__subsys__.c+, where +__subsys__+ is your
5023 .path:{probes/lttng-probe-my-subsys.c}
5025 #include <linux/module.h>
5026 #include "../lttng-tracer.h"
5029 * Build-time verification of mismatch between mainline
5030 * TRACE_EVENT() arguments and the LTTng-modules adaptation
5031 * layer LTTNG_TRACEPOINT_EVENT() arguments.
5033 #include <trace/events/my_subsys.h>
5035 /* Create LTTng tracepoint probes */
5036 #define LTTNG_PACKAGE_BUILD
5037 #define CREATE_TRACE_POINTS
5038 #define TRACE_INCLUDE_PATH ../instrumentation/events/lttng-module
5040 #include "../instrumentation/events/lttng-module/my_subsys.h"
5042 MODULE_LICENSE("GPL and additional rights");
5043 MODULE_AUTHOR("Your name <your-email>");
5044 MODULE_DESCRIPTION("LTTng my_subsys probes");
5045 MODULE_VERSION(__stringify(LTTNG_MODULES_MAJOR_VERSION) "."
5046 __stringify(LTTNG_MODULES_MINOR_VERSION) "."
5047 __stringify(LTTNG_MODULES_PATCHLEVEL_VERSION)
5048 LTTNG_MODULES_EXTRAVERSION);
5052 . Edit path:{probes/KBuild} and add your new kernel module object
5053 next to the existing ones:
5057 .path:{probes/KBuild}
5061 obj-m += lttng-probe-module.o
5062 obj-m += lttng-probe-power.o
5064 obj-m += lttng-probe-my-subsys.o
5070 . Build and install the LTTng kernel modules:
5075 $ make KERNELDIR=/path/to/linux
5076 # make modules_install && depmod -a
5080 Replace `/path/to/linux` with the path to the Linux source tree where
5081 you defined and used tracepoints with ftrace's `TRACE_EVENT()` macro.
5083 Note that you can also use the
5084 <<lttng-tracepoint-event-code,`LTTNG_TRACEPOINT_EVENT_CODE()` macro>>
5085 instead of `LTTNG_TRACEPOINT_EVENT()` to use custom local variables and
5086 C code that need to be executed before the event fields are recorded.
5088 The best way to learn how to use the previous LTTng-modules macros is to
5089 inspect the existing LTTng-modules tracepoint definitions in the
5090 dir:{instrumentation/events/lttng-module} header files. Compare them
5091 with the Linux kernel mainline versions in the
5092 dir:{include/trace/events} directory of the Linux source tree.
5096 [[lttng-tracepoint-event-code]]
5097 ===== Use custom C code to access the data for tracepoint fields
5099 Although we recommended to always use the
5100 <<lttng-adaptation-layer,`LTTNG_TRACEPOINT_EVENT()`>> macro to describe
5101 the arguments and fields of an LTTng-modules tracepoint when possible,
5102 sometimes you need a more complex process to access the data that the
5103 tracer records as event record fields. In other words, you need local
5104 variables and multiple C{nbsp}statements instead of simple
5105 argument-based expressions that you pass to the
5106 <<lttng-modules-tp-fields,`ctf_*()` macros of `TP_FIELDS()`>>.
5108 You can use the `LTTNG_TRACEPOINT_EVENT_CODE()` macro instead of
5109 `LTTNG_TRACEPOINT_EVENT()` to declare custom local variables and define
5110 a block of C{nbsp}code to be executed before LTTng records the fields.
5111 The structure of this macro is:
5114 .`LTTNG_TRACEPOINT_EVENT_CODE()` macro syntax.
5116 LTTNG_TRACEPOINT_EVENT_CODE(
5118 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5119 * version for the following three macro parameters:
5122 TP_PROTO(int my_int, const char *my_string),
5123 TP_ARGS(my_int, my_string),
5125 /* Declarations of custom local variables */
5128 unsigned long b = 0;
5129 const char *name = "(undefined)";
5130 struct my_struct *my_struct;
5134 * Custom code which uses both tracepoint arguments
5135 * (in TP_ARGS()) and local variables (in TP_locvar()).
5137 * Local variables are actually members of a structure pointed
5138 * to by the special variable tp_locvar.
5142 tp_locvar->a = my_int + 17;
5143 tp_locvar->my_struct = get_my_struct_at(tp_locvar->a);
5144 tp_locvar->b = my_struct_compute_b(tp_locvar->my_struct);
5145 tp_locvar->name = my_struct_get_name(tp_locvar->my_struct);
5146 put_my_struct(tp_locvar->my_struct);
5155 * Format identical to the LTTNG_TRACEPOINT_EVENT()
5156 * version for this, except that tp_locvar members can be
5157 * used in the argument expression parameters of
5158 * the ctf_*() macros.
5161 ctf_integer(unsigned long, my_struct_b, tp_locvar->b)
5162 ctf_integer(int, my_struct_a, tp_locvar->a)
5163 ctf_string(my_string_field, my_string)
5164 ctf_string(my_struct_name, tp_locvar->name)
5169 IMPORTANT: The C code defined in `TP_code()` must not have any side
5170 effects when executed. In particular, the code must not allocate
5171 memory or get resources without deallocating this memory or putting
5172 those resources afterwards.
5175 [[instrumenting-linux-kernel-tracing]]
5176 ==== Load and unload a custom probe kernel module
5178 You must load a <<lttng-adaptation-layer,created LTTng-modules probe
5179 kernel module>> in the kernel before it can emit LTTng events.
5181 To load the default probe kernel modules and a custom probe kernel
5184 * Use the opt:lttng-sessiond(8):--extra-kmod-probes option to give extra
5185 probe modules to load when starting a root <<lttng-sessiond,session
5189 .Load the `my_subsys`, `usb`, and the default probe modules.
5193 # lttng-sessiond --extra-kmod-probes=my_subsys,usb
5198 You only need to pass the subsystem name, not the whole kernel module
5201 To load _only_ a given custom probe kernel module:
5203 * Use the opt:lttng-sessiond(8):--kmod-probes option to give the probe
5204 modules to load when starting a root session daemon:
5207 .Load only the `my_subsys` and `usb` probe modules.
5211 # lttng-sessiond --kmod-probes=my_subsys,usb
5216 To confirm that a probe module is loaded:
5223 $ lsmod | grep lttng_probe_usb
5227 To unload the loaded probe modules:
5229 * Kill the session daemon with `SIGTERM`:
5234 # pkill lttng-sessiond
5238 You can also use man:modprobe(8)'s `--remove` option if the session
5239 daemon terminates abnormally.
5242 [[controlling-tracing]]
5245 Once an application or a Linux kernel is
5246 <<instrumenting,instrumented>> for LTTng tracing,
5249 This section is divided in topics on how to use the various
5250 <<plumbing,components of LTTng>>, in particular the <<lttng-cli,cmd:lttng
5251 command-line tool>>, to _control_ the LTTng daemons and tracers.
5253 NOTE: In the following subsections, we refer to an man:lttng(1) command
5254 using its man page name. For example, instead of _Run the `create`
5255 command to..._, we use _Run the man:lttng-create(1) command to..._.
5259 === Start a session daemon
5261 In some situations, you need to run a <<lttng-sessiond,session daemon>>
5262 (man:lttng-sessiond(8)) _before_ you can use the man:lttng(1)
5265 You will see the following error when you run a command while no session
5269 Error: No session daemon is available
5272 The only command that automatically runs a session daemon is
5273 man:lttng-create(1), which you use to
5274 <<creating-destroying-tracing-sessions,create a tracing session>>. While
5275 this is most of the time the first operation that you do, sometimes it's
5276 not. Some examples are:
5278 * <<list-instrumentation-points,List the available instrumentation points>>.
5279 * <<saving-loading-tracing-session,Load a tracing session configuration>>.
5281 [[tracing-group]] Each Unix user must have its own running session
5282 daemon to trace user applications. The session daemon that the root user
5283 starts is the only one allowed to control the LTTng kernel tracer. Users
5284 that are part of the _tracing group_ can control the root session
5285 daemon. The default tracing group name is `tracing`; you can set it to
5286 something else with the opt:lttng-sessiond(8):--group option when you
5287 start the root session daemon.
5289 To start a user session daemon:
5291 * Run man:lttng-sessiond(8):
5296 $ lttng-sessiond --daemonize
5300 To start the root session daemon:
5302 * Run man:lttng-sessiond(8) as the root user:
5307 # lttng-sessiond --daemonize
5311 In both cases, remove the opt:lttng-sessiond(8):--daemonize option to
5312 start the session daemon in foreground.
5314 To stop a session daemon, use man:kill(1) on its process ID (standard
5317 Note that some Linux distributions could manage the LTTng session daemon
5318 as a service. In this case, you should use the service manager to
5319 start, restart, and stop session daemons.
5322 [[creating-destroying-tracing-sessions]]
5323 === Create and destroy a tracing session
5325 Almost all the LTTng control operations happen in the scope of
5326 a <<tracing-session,tracing session>>, which is the dialogue between the
5327 <<lttng-sessiond,session daemon>> and you.
5329 To create a tracing session with a generated name:
5331 * Use the man:lttng-create(1) command:
5340 The created tracing session's name is `auto` followed by the
5343 To create a tracing session with a specific name:
5345 * Use the optional argument of the man:lttng-create(1) command:
5350 $ lttng create my-session
5354 Replace `my-session` with the specific tracing session name.
5356 LTTng appends the creation date to the created tracing session's name.
5358 LTTng writes the traces of a tracing session in
5359 +$LTTNG_HOME/lttng-trace/__name__+ by default, where +__name__+ is the
5360 name of the tracing session. Note that the env:LTTNG_HOME environment
5361 variable defaults to `$HOME` if not set.
5363 To output LTTng traces to a non-default location:
5365 * Use the opt:lttng-create(1):--output option of the man:lttng-create(1) command:
5370 $ lttng create my-session --output=/tmp/some-directory
5374 You may create as many tracing sessions as you wish.
5376 To list all the existing tracing sessions for your Unix user:
5378 * Use the man:lttng-list(1) command:
5387 When you create a tracing session, it is set as the _current tracing
5388 session_. The following man:lttng(1) commands operate on the current
5389 tracing session when you don't specify one:
5391 [role="list-3-cols"]
5407 To change the current tracing session:
5409 * Use the man:lttng-set-session(1) command:
5414 $ lttng set-session new-session
5418 Replace `new-session` by the name of the new current tracing session.
5420 When you are done tracing in a given tracing session, you can destroy
5421 it. This operation frees the resources taken by the tracing session
5422 to destroy; it does not destroy the trace data that LTTng wrote for
5423 this tracing session.
5425 To destroy the current tracing session:
5427 * Use the man:lttng-destroy(1) command:
5437 [[list-instrumentation-points]]
5438 === List the available instrumentation points
5440 The <<lttng-sessiond,session daemon>> can query the running instrumented
5441 user applications and the Linux kernel to get a list of available
5442 instrumentation points. For the Linux kernel <<domain,tracing domain>>,
5443 they are tracepoints and system calls. For the user space tracing
5444 domain, they are tracepoints. For the other tracing domains, they are
5447 To list the available instrumentation points:
5449 * Use the man:lttng-list(1) command with the requested tracing domain's
5453 * opt:lttng-list(1):--kernel: Linux kernel tracepoints (your Unix user
5454 must be a root user, or it must be a member of the
5455 <<tracing-group,tracing group>>).
5456 * opt:lttng-list(1):--kernel with opt:lttng-list(1):--syscall: Linux
5457 kernel system calls (your Unix user must be a root user, or it must be
5458 a member of the tracing group).
5459 * opt:lttng-list(1):--userspace: user space tracepoints.
5460 * opt:lttng-list(1):--jul: `java.util.logging` loggers.
5461 * opt:lttng-list(1):--log4j: Apache log4j loggers.
5462 * opt:lttng-list(1):--python: Python loggers.
5465 .List the available user space tracepoints.
5469 $ lttng list --userspace
5473 .List the available Linux kernel system call tracepoints.
5477 $ lttng list --kernel --syscall
5482 [[enabling-disabling-events]]
5483 === Create and enable an event rule
5485 Once you <<creating-destroying-tracing-sessions,create a tracing
5486 session>>, you can create <<event,event rules>> with the
5487 man:lttng-enable-event(1) command.
5489 You specify each condition with a command-line option. The available
5490 condition options are shown in the following table.
5492 [role="growable",cols="asciidoc,asciidoc,default"]
5493 .Condition command-line options for the man:lttng-enable-event(1) command.
5495 |Option |Description |Applicable tracing domains
5501 . +--probe=__ADDR__+
5502 . +--function=__ADDR__+
5505 Instead of using the default _tracepoint_ instrumentation type, use:
5507 . A Linux system call.
5508 . A Linux https://lwn.net/Articles/132196/[KProbe] (symbol or address).
5509 . The entry and return points of a Linux function (symbol or address).
5513 |First positional argument.
5516 Tracepoint or system call name. In the case of a Linux KProbe or
5517 function, this is a custom name given to the event rule. With the
5518 JUL, log4j, and Python domains, this is a logger name.
5520 With a tracepoint, logger, or system call name, the last character
5521 can be `*` to match anything that remains.
5528 . +--loglevel=__LEVEL__+
5529 . +--loglevel-only=__LEVEL__+
5532 . Match only tracepoints or log statements with a logging level at
5533 least as severe as +__LEVEL__+.
5534 . Match only tracepoints or log statements with a logging level
5535 equal to +__LEVEL__+.
5537 See man:lttng-enable-event(1) for the list of available logging level
5540 |User space, JUL, log4j, and Python.
5542 |+--exclude=__EXCLUSIONS__+
5545 When you use a `*` character at the end of the tracepoint or logger
5546 name (first positional argument), exclude the specific names in the
5547 comma-delimited list +__EXCLUSIONS__+.
5550 User space, JUL, log4j, and Python.
5552 |+--filter=__EXPR__+
5555 Match only events which satisfy the expression +__EXPR__+.
5557 See man:lttng-enable-event(1) to learn more about the syntax of a
5564 You attach an event rule to a <<channel,channel>> on creation. If you do
5565 not specify the channel with the opt:lttng-enable-event(1):--channel
5566 option, and if the event rule to create is the first in its
5567 <<domain,tracing domain>> for a given tracing session, then LTTng
5568 creates a _default channel_ for you. This default channel is reused in
5569 subsequent invocations of the man:lttng-enable-event(1) command for the
5570 same tracing domain.
5572 An event rule is always enabled at creation time.
5574 The following examples show how you can combine the previous
5575 command-line options to create simple to more complex event rules.
5577 .Create an event rule targetting a Linux kernel tracepoint (default channel).
5581 $ lttng enable-event --kernel sched_switch
5585 .Create an event rule matching four Linux kernel system calls (default channel).
5589 $ lttng enable-event --kernel --syscall open,write,read,close
5593 .Create event rules matching tracepoints with filter expressions (default channel).
5597 $ lttng enable-event --kernel sched_switch --filter='prev_comm == "bash"'
5602 $ lttng enable-event --kernel --all \
5603 --filter='$ctx.tid == 1988 || $ctx.tid == 1534'
5608 $ lttng enable-event --jul my_logger \
5609 --filter='$app.retriever:cur_msg_id > 3'
5612 IMPORTANT: Make sure to always quote the filter string when you
5613 use man:lttng(1) from a shell.
5616 .Create an event rule matching any user space tracepoint of a given tracepoint provider with a log level range (default channel).
5620 $ lttng enable-event --userspace my_app:'*' --loglevel=TRACE_INFO
5623 IMPORTANT: Make sure to always quote the wildcard character when you
5624 use man:lttng(1) from a shell.
5627 .Create an event rule matching multiple Python loggers with a wildcard and with exclusions (default channel).
5631 $ lttng enable-event --python my-app.'*' \
5632 --exclude='my-app.module,my-app.hello'
5636 .Create an event rule matching any Apache log4j logger with a specific log level (default channel).
5640 $ lttng enable-event --log4j --all --loglevel-only=LOG4J_WARN
5644 .Create an event rule attached to a specific channel matching a specific user space tracepoint provider and tracepoint.
5648 $ lttng enable-event --userspace my_app:my_tracepoint --channel=my-channel
5652 The event rules of a given channel form a whitelist: as soon as an
5653 emitted event passes one of them, LTTng can record the event. For
5654 example, an event named `my_app:my_tracepoint` emitted from a user space
5655 tracepoint with a `TRACE_ERROR` log level passes both of the following
5660 $ lttng enable-event --userspace my_app:my_tracepoint
5661 $ lttng enable-event --userspace my_app:my_tracepoint \
5662 --loglevel=TRACE_INFO
5665 The second event rule is redundant: the first one includes
5669 [[disable-event-rule]]
5670 === Disable an event rule
5672 To disable an event rule that you <<enabling-disabling-events,created>>
5673 previously, use the man:lttng-disable-event(1) command. This command
5674 disables _all_ the event rules (of a given tracing domain and channel)
5675 which match an instrumentation point. The other conditions are not
5676 supported as of LTTng{nbsp}{revision}.
5678 The LTTng tracer does not record an emitted event which passes
5679 a _disabled_ event rule.
5681 .Disable an event rule matching a Python logger (default channel).
5685 $ lttng disable-event --python my-logger
5689 .Disable an event rule matching all `java.util.logging` loggers (default channel).
5693 $ lttng disable-event --jul '*'
5697 .Disable _all_ the event rules of the default channel.
5699 The opt:lttng-disable-event(1):--all-events option is not, like the
5700 opt:lttng-enable-event(1):--all option of man:lttng-enable-event(1), the
5701 equivalent of the event name `*` (wildcard): it disables _all_ the event
5702 rules of a given channel.
5706 $ lttng disable-event --jul --all-events
5710 NOTE: You cannot delete an event rule once you create it.
5714 === Get the status of a tracing session
5716 To get the status of the current tracing session, that is, its
5717 parameters, its channels, event rules, and their attributes:
5719 * Use the man:lttng-status(1) command:
5729 To get the status of any tracing session:
5731 * Use the man:lttng-list(1) command with the tracing session's name:
5736 $ lttng list my-session
5740 Replace `my-session` with the desired tracing session's name.
5743 [[basic-tracing-session-control]]
5744 === Start and stop a tracing session
5746 Once you <<creating-destroying-tracing-sessions,create a tracing
5748 <<enabling-disabling-events,create one or more event rules>>,
5749 you can start and stop the tracers for this tracing session.
5751 To start tracing in the current tracing session:
5753 * Use the man:lttng-start(1) command:
5762 LTTng is very flexible: you can launch user applications before
5763 or after the you start the tracers. The tracers only record the events
5764 if they pass enabled event rules and if they occur while the tracers are
5767 To stop tracing in the current tracing session:
5769 * Use the man:lttng-stop(1) command:
5778 If there were <<channel-overwrite-mode-vs-discard-mode,lost event
5779 records>> or lost sub-buffers since the last time you ran
5780 man:lttng-start(1), warnings are printed when you run the
5781 man:lttng-stop(1) command.
5784 [[enabling-disabling-channels]]
5785 === Create a channel
5787 Once you create a tracing session, you can create a <<channel,channel>>
5788 with the man:lttng-enable-channel(1) command.
5790 Note that LTTng automatically creates a default channel when, for a
5791 given <<domain,tracing domain>>, no channels exist and you
5792 <<enabling-disabling-events,create>> the first event rule. This default
5793 channel is named `channel0` and its attributes are set to reasonable
5794 values. Therefore, you only need to create a channel when you need
5795 non-default attributes.
5797 You specify each non-default channel attribute with a command-line
5798 option when you use the man:lttng-enable-channel(1) command. The
5799 available command-line options are:
5801 [role="growable",cols="asciidoc,asciidoc"]
5802 .Command-line options for the man:lttng-enable-channel(1) command.
5804 |Option |Description
5810 <<channel-overwrite-mode-vs-discard-mode,event loss mode>> instead of
5811 the default _discard_ mode.
5813 |`--buffers-pid` (user space tracing domain only)
5816 Use the per-process <<channel-buffering-schemes,buffering scheme>>
5817 instead of the default per-user buffering scheme.
5819 |+--subbuf-size=__SIZE__+
5822 Allocate sub-buffers of +__SIZE__+ bytes (power of two), for each CPU,
5823 either for each Unix user (default), or for each instrumented process.
5825 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
5827 |+--num-subbuf=__COUNT__+
5830 Allocate +__COUNT__+ sub-buffers (power of two), for each CPU, either
5831 for each Unix user (default), or for each instrumented process.
5833 See <<channel-subbuf-size-vs-subbuf-count,Sub-buffer count and size>>.
5835 |+--tracefile-size=__SIZE__+
5838 Set the maximum size of each trace file that this channel writes within
5839 a stream to +__SIZE__+ bytes instead of no maximum.
5841 See <<tracefile-rotation,Trace file count and size>>.
5843 |+--tracefile-count=__COUNT__+
5846 Limit the number of trace files that this channel creates to
5847 +__COUNT__+ channels instead of no limit.
5849 See <<tracefile-rotation,Trace file count and size>>.
5851 |+--switch-timer=__PERIODUS__+
5854 Set the <<channel-switch-timer,switch timer period>>
5855 to +__PERIODUS__+{nbsp}µs.
5857 |+--read-timer=__PERIODUS__+
5860 Set the <<channel-read-timer,read timer period>>
5861 to +__PERIODUS__+{nbsp}µs.
5863 |+--output=__TYPE__+ (Linux kernel tracing domain only)
5866 Set the channel's output type to +__TYPE__+, either `mmap` or `splice`.
5870 You can only create a channel in the Linux kernel and user space
5871 <<domain,tracing domains>>: other tracing domains have their own channel
5872 created on the fly when <<enabling-disabling-events,creating event
5877 Because of a current LTTng limitation, you must create all channels
5878 _before_ you <<basic-tracing-session-control,start tracing>> in a given
5879 tracing session, that is, before the first time you run
5882 Since LTTng automatically creates a default channel when you use the
5883 man:lttng-enable-event(1) command with a specific tracing domain, you
5884 cannot, for example, create a Linux kernel event rule, start tracing,
5885 and then create a user space event rule, because no user space channel
5886 exists yet and it's too late to create one.
5888 For this reason, make sure to configure your channels properly
5889 before starting the tracers for the first time!
5892 The following examples show how you can combine the previous
5893 command-line options to create simple to more complex channels.
5895 .Create a Linux kernel channel with default attributes.
5899 $ lttng enable-channel --kernel my-channel
5903 .Create a user space channel with 4 sub-buffers or 1{nbsp}MiB each, per CPU, per instrumented process.
5907 $ lttng enable-channel --userspace --num-subbuf=4 --subbuf-size=1M \
5908 --buffers-pid my-channel
5912 .Create a Linux kernel channel which rotates 8 trace files of 4{nbsp}MiB each for each stream
5916 $ lttng enable-channel --kernel --tracefile-count=8 \
5917 --tracefile-size=4194304 my-channel
5921 .Create a user space channel in overwrite (or _flight recorder_) mode.
5925 $ lttng enable-channel --userspace --overwrite my-channel
5929 You can <<enabling-disabling-events,create>> the same event rule in
5930 two different channels:
5934 $ lttng enable-event --userspace --channel=my-channel app:tp
5935 $ lttng enable-event --userspace --channel=other-channel app:tp
5938 If both channels are enabled, when a tracepoint named `app:tp` is
5939 reached, LTTng records two events, one for each channel.
5943 === Disable a channel
5945 To disable a specific channel that you <<enabling-disabling-channels,created>>
5946 previously, use the man:lttng-disable-channel(1) command.
5948 .Disable a specific Linux kernel channel.
5952 $ lttng disable-channel --kernel my-channel
5956 The state of a channel precedes the individual states of event rules
5957 attached to it: event rules which belong to a disabled channel, even if
5958 they are enabled, are also considered disabled.
5962 === Add context fields to a channel
5964 Event record fields in trace files provide important information about
5965 events that occured previously, but sometimes some external context may
5966 help you solve a problem faster. Examples of context fields are:
5968 * The **process ID**, **thread ID**, **process name**, and
5969 **process priority** of the thread in which the event occurs.
5970 * The **hostname** of the system on which the event occurs.
5971 * The current values of many possible **performance counters** using
5973 ** CPU cycles, stalled cycles, idle cycles, and the other cycle types.
5975 ** Branch instructions, misses, and loads.
5977 * Any context defined at the application level (supported for the
5978 JUL and log4j <<domain,tracing domains>>).
5980 To get the full list of available context fields, see
5981 `lttng add-context --list`. Some context fields are reserved for a
5982 specific <<domain,tracing domain>> (Linux kernel or user space).
5984 You add context fields to <<channel,channels>>. All the events
5985 that a channel with added context fields records contain those fields.
5987 To add context fields to one or all the channels of a given tracing
5990 * Use the man:lttng-add-context(1) command.
5992 .Add context fields to all the channels of the current tracing session.
5994 The following command line adds the virtual process identifier and
5995 the per-thread CPU cycles count fields to all the user space channels
5996 of the current tracing session.
6000 $ lttng add-context --userspace --type=vpid --type=perf:thread:cpu-cycles
6004 .Add a context field to a specific channel.
6006 The following command line adds the thread identifier context field
6007 to the Linux kernel channel named `my-channel` in the current
6012 $ lttng add-context --kernel --channel=my-channel --type=tid
6016 .Add an application-specific context field to a specific channel.
6018 The following command line adds the `cur_msg_id` context field of the
6019 `retriever` context retriever for all the instrumented
6020 <<java-application,Java applications>> recording <<event,event records>>
6021 in the channel named `my-channel`:
6025 $ lttng add-context --kernel --channel=my-channel \
6026 --type='$app:retriever:cur_msg_id'
6029 IMPORTANT: Make sure to always quote the `$` character when you
6030 use man:lttng-add-context(1) from a shell.
6033 NOTE: You cannot remove context fields from a channel once you add it.
6038 === Track process IDs
6040 It's often useful to allow only specific process IDs (PIDs) to emit
6041 events. For example, you may wish to record all the system calls made by
6042 a given process (à la http://linux.die.net/man/1/strace[strace]).
6044 The man:lttng-track(1) and man:lttng-untrack(1) commands serve this
6045 purpose. Both commands operate on a whitelist of process IDs. You _add_
6046 entries to this whitelist with the man:lttng-track(1) command and remove
6047 entries with the man:lttng-untrack(1) command. Any process which has one
6048 of the PIDs in the whitelist is allowed to emit LTTng events which pass
6049 an enabled <<event,event rule>>.
6051 NOTE: The PID tracker tracks the _numeric process IDs_. Should a
6052 process with a given tracked ID exit and another process be given this
6053 ID, then the latter would also be allowed to emit events.
6055 .Track and untrack process IDs.
6057 For the sake of the following example, assume the target system has 16
6061 <<creating-destroying-tracing-sessions,create a tracing session>>,
6062 the whitelist contains all the possible PIDs:
6065 .All PIDs are tracked.
6066 image::track-all.png[]
6068 When the whitelist is full and you use the man:lttng-track(1) command to
6069 specify some PIDs to track, LTTng first clears the whitelist, then it
6070 tracks the specific PIDs. After:
6074 $ lttng track --pid=3,4,7,10,13
6080 .PIDs 3, 4, 7, 10, and 13 are tracked.
6081 image::track-3-4-7-10-13.png[]
6083 You can add more PIDs to the whitelist afterwards:
6087 $ lttng track --pid=1,15,16
6093 .PIDs 1, 15, and 16 are added to the whitelist.
6094 image::track-1-3-4-7-10-13-15-16.png[]
6096 The man:lttng-untrack(1) command removes entries from the PID tracker's
6097 whitelist. Given the previous example, the following command:
6101 $ lttng untrack --pid=3,7,10,13
6104 leads to this whitelist:
6107 .PIDs 3, 7, 10, and 13 are removed from the whitelist.
6108 image::track-1-4-15-16.png[]
6110 LTTng can track all possible PIDs again using the
6111 opt:lttng-track(1):--all option:
6115 $ lttng track --pid --all
6118 The result is, again:
6121 .All PIDs are tracked.
6122 image::track-all.png[]
6125 .Track only specific PIDs
6127 A very typical use case with PID tracking is to start with an empty
6128 whitelist, then <<basic-tracing-session-control,start the tracers>>, and
6129 then add PIDs manually while tracers are active. You can accomplish this
6130 by using the opt:lttng-untrack(1):--all option of the
6131 man:lttng-untrack(1) command to clear the whitelist after you
6132 <<creating-destroying-tracing-sessions,create a tracing session>>:
6136 $ lttng untrack --pid --all
6142 .No PIDs are tracked.
6143 image::untrack-all.png[]
6145 If you trace with this whitelist configuration, the tracer records no
6146 events for this <<domain,tracing domain>> because no processes are
6147 tracked. You can use the man:lttng-track(1) command as usual to track
6148 specific PIDs, for example:
6152 $ lttng track --pid=6,11
6158 .PIDs 6 and 11 are tracked.
6159 image::track-6-11.png[]
6164 [[saving-loading-tracing-session]]
6165 === Save and load tracing session configurations
6167 Configuring a <<tracing-session,tracing session>> can be long. Some of
6168 the tasks involved are:
6170 * <<enabling-disabling-channels,Create channels>> with
6171 specific attributes.
6172 * <<adding-context,Add context fields>> to specific channels.
6173 * <<enabling-disabling-events,Create event rules>> with specific log
6174 level and filter conditions.
6176 If you use LTTng to solve real world problems, chances are you have to
6177 record events using the same tracing session setup over and over,
6178 modifying a few variables each time in your instrumented program
6179 or environment. To avoid constant tracing session reconfiguration,
6180 the man:lttng(1) command-line tool can save and load tracing session
6181 configurations to/from XML files.
6183 To save a given tracing session configuration:
6185 * Use the man:lttng-save(1) command:
6190 $ lttng save my-session
6194 Replace `my-session` with the name of the tracing session to save.
6196 LTTng saves tracing session configurations to
6197 dir:{$LTTNG_HOME/.lttng/sessions} by default. Note that the
6198 env:LTTNG_HOME environment variable defaults to `$HOME` if not set. Use
6199 the opt:lttng-save(1):--output-path option to change this destination
6202 LTTng saves all configuration parameters, for example:
6204 * The tracing session name.
6205 * The trace data output path.
6206 * The channels with their state and all their attributes.
6207 * The context fields you added to channels.
6208 * The event rules with their state, log level and filter conditions.
6210 To load a tracing session:
6212 * Use the man:lttng-load(1) command:
6217 $ lttng load my-session
6221 Replace `my-session` with the name of the tracing session to load.
6223 When LTTng loads a configuration, it restores your saved tracing session
6224 as if you just configured it manually.
6226 See man:lttng(1) for the complete list of command-line options. You
6227 can also save and load all many sessions at a time, and decide in which
6228 directory to output the XML files.
6231 [[sending-trace-data-over-the-network]]
6232 === Send trace data over the network
6234 LTTng can send the recorded trace data to a remote system over the
6235 network instead of writing it to the local file system.
6237 To send the trace data over the network:
6239 . On the _remote_ system (which can also be the target system),
6240 start an LTTng <<lttng-relayd,relay daemon>> (man:lttng-relayd(8)):
6249 . On the _target_ system, create a tracing session configured to
6250 send trace data over the network:
6255 $ lttng create my-session --set-url=net://remote-system
6259 Replace `remote-system` by the host name or IP address of the
6260 remote system. See man:lttng-create(1) for the exact URL format.
6262 . On the target system, use the man:lttng(1) command-line tool as usual.
6263 When tracing is active, the target's consumer daemon sends sub-buffers
6264 to the relay daemon running on the remote system instead of flushing
6265 them to the local file system. The relay daemon writes the received
6266 packets to the local file system.
6268 The relay daemon writes trace files to
6269 +$LTTNG_HOME/lttng-traces/__hostname__/__session__+ by default, where
6270 +__hostname__+ is the host name of the target system and +__session__+
6271 is the tracing session name. Note that the env:LTTNG_HOME environment
6272 variable defaults to `$HOME` if not set. Use the
6273 opt:lttng-relayd(8):--output option of man:lttng-relayd(8) to write
6274 trace files to another base directory.
6279 === View events as LTTng emits them (noch:{LTTng} live)
6281 LTTng live is a network protocol implemented by the <<lttng-relayd,relay
6282 daemon>> (man:lttng-relayd(8)) to allow compatible trace viewers to
6283 display events as LTTng emits them on the target system while tracing is
6286 The relay daemon creates a _tee_: it forwards the trace data to both
6287 the local file system and to connected live viewers:
6290 .The relay daemon creates a _tee_, forwarding the trace data to both trace files and a connected live viewer.
6295 . On the _target system_, create a <<tracing-session,tracing session>>
6301 $ lttng create my-session --live
6305 This spawns a local relay daemon.
6307 . Start the live viewer and configure it to connect to the relay
6308 daemon. For example, with http://diamon.org/babeltrace[Babeltrace]:
6313 $ babeltrace --input-format=lttng-live
6314 net://localhost/host/hostname/my-session
6321 * `hostname` with the host name of the target system.
6322 * `my-session` with the name of the tracing session to view.
6325 . Configure the tracing session as usual with the man:lttng(1)
6326 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6328 You can list the available live tracing sessions with Babeltrace:
6332 $ babeltrace --input-format=lttng-live net://localhost
6335 You can start the relay daemon on another system. In this case, you need
6336 to specify the relay daemon's URL when you create the tracing session
6337 with the opt:lttng-create(1):--set-url option. You also need to replace
6338 `localhost` in the procedure above with the host name of the system on
6339 which the relay daemon is running.
6341 See man:lttng-create(1) and man:lttng-relayd(8) for the complete list of
6342 command-line options.
6346 [[taking-a-snapshot]]
6347 === Take a snapshot of the current sub-buffers of a tracing session
6349 The normal behavior of LTTng is to append full sub-buffers to growing
6350 trace data files. This is ideal to keep a full history of the events
6351 that occurred on the target system, but it can
6352 represent too much data in some situations. For example, you may wish
6353 to trace your application continuously until some critical situation
6354 happens, in which case you only need the latest few recorded
6355 events to perform the desired analysis, not multi-gigabyte trace files.
6357 With the man:lttng-snapshot(1) command, you can take a snapshot of the
6358 current sub-buffers of a given <<tracing-session,tracing session>>.
6359 LTTng can write the snapshot to the local file system or send it over
6364 . Create a tracing session in _snapshot mode_:
6369 $ lttng create my-session --snapshot
6373 The <<channel-overwrite-mode-vs-discard-mode,event loss mode>> of
6374 <<channel,channels>> created in this mode is automatically set to
6375 _overwrite_ (flight recorder mode).
6377 . Configure the tracing session as usual with the man:lttng(1)
6378 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6380 . **Optional**: When you need to take a snapshot,
6381 <<basic-tracing-session-control,stop tracing>>.
6383 You can take a snapshot when the tracers are active, but if you stop
6384 them first, you are sure that the data in the sub-buffers does not
6385 change before you actually take the snapshot.
6392 $ lttng snapshot record --name=my-first-snapshot
6396 LTTng writes the current sub-buffers of all the current tracing
6397 session's channels to trace files on the local file system. Those trace
6398 files have `my-first-snapshot` in their name.
6400 There is no difference between the format of a normal trace file and the
6401 format of a snapshot: viewers of LTTng traces also support LTTng
6404 By default, LTTng writes snapshot files to the path shown by
6405 `lttng snapshot list-output`. You can change this path or decide to send
6406 snapshots over the network using either:
6408 . An output path or URL that you specify when you create the
6410 . An snapshot output path or URL that you add using
6411 `lttng snapshot add-output`
6412 . An output path or URL that you provide directly to the
6413 `lttng snapshot record` command.
6415 Method 3 overrides method 2, which overrides method 1. When you
6416 specify a URL, a relay daemon must listen on a remote system (see
6417 <<sending-trace-data-over-the-network,Send trace data over the network>>).
6422 === Use the machine interface
6424 With any command of the man:lttng(1) command-line tool, you can set the
6425 opt:lttng(1):--mi option to `xml` (before the command name) to get an
6426 XML machine interface output, for example:
6430 $ lttng --mi=xml enable-event --kernel --syscall open
6433 A schema definition (XSD) is
6434 https://github.com/lttng/lttng-tools/blob/stable-2.8/src/common/mi-lttng-3.0.xsd[available]
6435 to ease the integration with external tools as much as possible.
6439 [[metadata-regenerate]]
6440 === Regenerate the metadata of an LTTng trace
6442 An LTTng trace, which is a http://diamon.org/ctf[CTF] trace, has both
6443 data stream files and a metadata file. This metadata file contains,
6444 amongst other things, information about the offset of the clock sources
6445 used to timestamp <<event,event records>> when tracing.
6447 If, once a <<tracing-session,tracing session>> is
6448 <<basic-tracing-session-control,started>>, a major
6449 https://en.wikipedia.org/wiki/Network_Time_Protocol[NTP] correction
6450 happens, the trace's clock offset also needs to be updated. You
6451 can use the man:lttng-metadata(1) command to do so.
6453 The main use case of this command is to allow a system to boot with
6454 an incorrect wall time and trace it with LTTng before its wall time
6455 is corrected. Once the system is known to be in a state where its
6456 wall time is correct, it can run `lttng metadata regenerate`.
6458 To regenerate the metadata of an LTTng trace:
6460 * Use the `regenerate` action of the man:lttng-metadata(1) command:
6465 $ lttng metadata regenerate
6471 `lttng metadata regenerate` has the following limitations:
6473 * Tracing session <<creating-destroying-tracing-sessions,created>>
6475 * User space <<channel,channels>>, if any, using
6476 <<channel-buffering-schemes,per-user buffering>>.
6481 [[persistent-memory-file-systems]]
6482 === Record trace data on persistent memory file systems
6484 https://en.wikipedia.org/wiki/Non-volatile_random-access_memory[Non-volatile random-access memory]
6485 (NVRAM) is random-access memory that retains its information when power
6486 is turned off (non-volatile). Systems with such memory can store data
6487 structures in RAM and retrieve them after a reboot, without flushing
6488 to typical _storage_.
6490 Linux supports NVRAM file systems thanks to either
6491 http://pramfs.sourceforge.net/[PRAMFS] or
6492 https://www.kernel.org/doc/Documentation/filesystems/dax.txt[DAX]{nbsp}+{nbsp}http://lkml.iu.edu/hypermail/linux/kernel/1504.1/03463.html[pmem]
6493 (requires Linux 4.1+).
6495 This section does not describe how to operate such file systems;
6496 we assume that you have a working persistent memory file system.
6498 When you create a <<tracing-session,tracing session>>, you can specify
6499 the path of the shared memory holding the sub-buffers. If you specify a
6500 location on an NVRAM file system, then you can retrieve the latest
6501 recorded trace data when the system reboots after a crash.
6503 To record trace data on a persistent memory file system and retrieve the
6504 trace data after a system crash:
6506 . Create a tracing session with a sub-buffer shared memory path located
6507 on an NVRAM file system:
6512 $ lttng create my-session --shm-path=/path/to/shm
6516 . Configure the tracing session as usual with the man:lttng(1)
6517 command-line tool, and <<basic-tracing-session-control,start tracing>>.
6519 . After a system crash, use the man:lttng-crash(1) command-line tool to
6520 view the trace data recorded on the NVRAM file system:
6525 $ lttng-crash /path/to/shm
6529 The binary layout of the ring buffer files is not exactly the same as
6530 the trace files layout. This is why you need to use man:lttng-crash(1)
6531 instead of your preferred trace viewer directly.
6533 To convert the ring buffer files to LTTng trace files:
6535 * Use the opt:lttng-crash(1):--extract option of man:lttng-crash(1):
6540 $ lttng-crash --extract=/path/to/trace /path/to/shm
6548 [[lttng-modules-ref]]
6549 === noch:{LTTng-modules}
6552 [[lttng-modules-tp-fields]]
6553 ==== Tracepoint fields macros (for `TP_FIELDS()`)
6555 [[tp-fast-assign]][[tp-struct-entry]]The available macros to define
6556 tracepoint fields, which must be listed within `TP_FIELDS()` in
6557 `LTTNG_TRACEPOINT_EVENT()`, are:
6559 [role="func-desc growable",cols="asciidoc,asciidoc"]
6560 .Available macros to define LTTng-modules tracepoint fields
6562 |Macro |Description and parameters
6565 +ctf_integer(__t__, __n__, __e__)+
6567 +ctf_integer_nowrite(__t__, __n__, __e__)+
6569 +ctf_user_integer(__t__, __n__, __e__)+
6571 +ctf_user_integer_nowrite(__t__, __n__, __e__)+
6573 Standard integer, displayed in base 10.
6576 Integer C type (`int`, `long`, `size_t`, ...).
6582 Argument expression.
6585 +ctf_integer_hex(__t__, __n__, __e__)+
6587 +ctf_user_integer_hex(__t__, __n__, __e__)+
6589 Standard integer, displayed in base 16.
6598 Argument expression.
6600 |+ctf_integer_oct(__t__, __n__, __e__)+
6602 Standard integer, displayed in base 8.
6611 Argument expression.
6614 +ctf_integer_network(__t__, __n__, __e__)+
6616 +ctf_user_integer_network(__t__, __n__, __e__)+
6618 Integer in network byte order (big-endian), displayed in base 10.
6627 Argument expression.
6630 +ctf_integer_network_hex(__t__, __n__, __e__)+
6632 +ctf_user_integer_network_hex(__t__, __n__, __e__)+
6634 Integer in network byte order, displayed in base 16.
6643 Argument expression.
6646 +ctf_string(__n__, __e__)+
6648 +ctf_string_nowrite(__n__, __e__)+
6650 +ctf_user_string(__n__, __e__)+
6652 +ctf_user_string_nowrite(__n__, __e__)+
6654 Null-terminated string; undefined behavior if +__e__+ is `NULL`.
6660 Argument expression.
6663 +ctf_array(__t__, __n__, __e__, __s__)+
6665 +ctf_array_nowrite(__t__, __n__, __e__, __s__)+
6667 +ctf_user_array(__t__, __n__, __e__, __s__)+
6669 +ctf_user_array_nowrite(__t__, __n__, __e__, __s__)+
6671 Statically-sized array of integers.
6674 Array element C type.
6680 Argument expression.
6686 +ctf_array_bitfield(__t__, __n__, __e__, __s__)+
6688 +ctf_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
6690 +ctf_user_array_bitfield(__t__, __n__, __e__, __s__)+
6692 +ctf_user_array_bitfield_nowrite(__t__, __n__, __e__, __s__)+
6694 Statically-sized array of bits.
6696 The type of +__e__+ must be an integer type. +__s__+ is the number
6697 of elements of such type in +__e__+, not the number of bits.
6700 Array element C type.
6706 Argument expression.
6712 +ctf_array_text(__t__, __n__, __e__, __s__)+
6714 +ctf_array_text_nowrite(__t__, __n__, __e__, __s__)+
6716 +ctf_user_array_text(__t__, __n__, __e__, __s__)+
6718 +ctf_user_array_text_nowrite(__t__, __n__, __e__, __s__)+
6720 Statically-sized array, printed as text.
6722 The string does not need to be null-terminated.
6725 Array element C type (always `char`).
6731 Argument expression.
6737 +ctf_sequence(__t__, __n__, __e__, __T__, __E__)+
6739 +ctf_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
6741 +ctf_user_sequence(__t__, __n__, __e__, __T__, __E__)+
6743 +ctf_user_sequence_nowrite(__t__, __n__, __e__, __T__, __E__)+
6745 Dynamically-sized array of integers.
6747 The type of +__E__+ must be unsigned.
6750 Array element C type.
6756 Argument expression.
6759 Length expression C type.
6765 +ctf_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
6767 +ctf_user_sequence_hex(__t__, __n__, __e__, __T__, __E__)+
6769 Dynamically-sized array of integers, displayed in base 16.
6771 The type of +__E__+ must be unsigned.
6774 Array element C type.
6780 Argument expression.
6783 Length expression C type.
6788 |+ctf_sequence_network(__t__, __n__, __e__, __T__, __E__)+
6790 Dynamically-sized array of integers in network byte order (big-endian),
6791 displayed in base 10.
6793 The type of +__E__+ must be unsigned.
6796 Array element C type.
6802 Argument expression.
6805 Length expression C type.
6811 +ctf_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
6813 +ctf_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
6815 +ctf_user_sequence_bitfield(__t__, __n__, __e__, __T__, __E__)+
6817 +ctf_user_sequence_bitfield_nowrite(__t__, __n__, __e__, __T__, __E__)+
6819 Dynamically-sized array of bits.
6821 The type of +__e__+ must be an integer type. +__s__+ is the number
6822 of elements of such type in +__e__+, not the number of bits.
6824 The type of +__E__+ must be unsigned.
6827 Array element C type.
6833 Argument expression.
6836 Length expression C type.
6842 +ctf_sequence_text(__t__, __n__, __e__, __T__, __E__)+
6844 +ctf_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
6846 +ctf_user_sequence_text(__t__, __n__, __e__, __T__, __E__)+
6848 +ctf_user_sequence_text_nowrite(__t__, __n__, __e__, __T__, __E__)+
6850 Dynamically-sized array, displayed as text.
6852 The string does not need to be null-terminated.
6854 The type of +__E__+ must be unsigned.
6856 The behaviour is undefined if +__e__+ is `NULL`.
6859 Sequence element C type (always `char`).
6865 Argument expression.
6868 Length expression C type.
6874 Use the `_user` versions when the argument expression, `e`, is
6875 a user space address. In the cases of `ctf_user_integer*()` and
6876 `ctf_user_float*()`, `&e` must be a user space address, thus `e` must
6879 The `_nowrite` versions omit themselves from the session trace, but are
6880 otherwise identical. This means the `_nowrite` fields won't be written
6881 in the recorded trace. Their primary purpose is to make some
6882 of the event context available to the
6883 <<enabling-disabling-events,event filters>> without having to
6884 commit the data to sub-buffers.
6890 Terms related to LTTng and to tracing in general:
6893 The http://diamon.org/babeltrace[Babeltrace] project, which includes
6894 the cmd:babeltrace command, some libraries, and Python bindings.
6896 <<channel-buffering-schemes,buffering scheme>>::
6897 A layout of sub-buffers applied to a given channel.
6899 <<channel,channel>>::
6900 An entity which is responsible for a set of ring buffers.
6902 <<event,Event rules>> are always attached to a specific channel.
6905 A reference of time for a tracer.
6907 <<lttng-consumerd,consumer daemon>>::
6908 A process which is responsible for consuming the full sub-buffers
6909 and write them to a file system or send them over the network.
6911 <<channel-overwrite-mode-vs-discard-mode,discard mode>>:: The event loss
6912 mode in which the tracer _discards_ new event records when there's no
6913 sub-buffer space left to store them.
6916 The consequence of the execution of an instrumentation
6917 point, like a tracepoint that you manually place in some source code,
6918 or a Linux kernel KProbe.
6920 An event is said to _occur_ at a specific time. Different actions can
6921 be taken upon the occurrence of an event, like record the event's payload
6924 <<channel-overwrite-mode-vs-discard-mode,event loss mode>>::
6925 The mechanism by which event records of a given channel are lost
6926 (not recorded) when there is no sub-buffer space left to store them.
6928 [[def-event-name]]event name::
6929 The name of an event, which is also the name of the event record.
6930 This is also called the _instrumentation point name_.
6933 A record, in a trace, of the payload of an event which occured.
6935 <<event,event rule>>::
6936 Set of conditions which must be satisfied for one or more occuring
6937 events to be recorded.
6939 `java.util.logging`::
6941 https://docs.oracle.com/javase/7/docs/api/java/util/logging/package-summary.html[core logging facilities].
6943 <<instrumenting,instrumentation>>::
6944 The use of LTTng probes to make a piece of software traceable.
6946 instrumentation point::
6947 A point in the execution path of a piece of software that, when
6948 reached by this execution, can emit an event.
6950 instrumentation point name::
6951 See _<<def-event-name,event name>>_.
6954 A http://logging.apache.org/log4j/1.2/[logging library] for Java
6955 developed by the Apache Software Foundation.
6958 Level of severity of a log statement or user space
6959 instrumentation point.
6962 The _Linux Trace Toolkit: next generation_ project.
6964 <<lttng-cli,cmd:lttng>>::
6965 A command-line tool provided by the LTTng-tools project which you
6966 can use to send and receive control messages to and from a
6970 The https://github.com/lttng/lttng-analyses[LTTng analyses] project,
6971 which is a set of analyzing programs that are used to obtain a
6972 higher level view of an LTTng trace.
6974 cmd:lttng-consumerd::
6975 The name of the consumer daemon program.
6978 A utility provided by the LTTng-tools project which can convert
6979 ring buffer files (usually
6980 <<persistent-memory-file-systems,saved on a persistent memory file system>>)
6983 LTTng Documentation::
6986 <<lttng-live,LTTng live>>::
6987 A communication protocol between the relay daemon and live viewers
6988 which makes it possible to see events "live", as they are received by
6991 <<lttng-modules,LTTng-modules>>::
6992 The https://github.com/lttng/lttng-modules[LTTng-modules] project,
6993 which contains the Linux kernel modules to make the Linux kernel
6994 instrumentation points available for LTTng tracing.
6997 The name of the relay daemon program.
6999 cmd:lttng-sessiond::
7000 The name of the session daemon program.
7003 The https://github.com/lttng/lttng-tools[LTTng-tools] project, which
7004 contains the various programs and libraries used to
7005 <<controlling-tracing,control tracing>>.
7007 <<lttng-ust,LTTng-UST>>::
7008 The https://github.com/lttng/lttng-ust[LTTng-UST] project, which
7009 contains libraries to instrument user applications.
7011 <<lttng-ust-agents,LTTng-UST Java agent>>::
7012 A Java package provided by the LTTng-UST project to allow the
7013 LTTng instrumentation of `java.util.logging` and Apache log4j 1.2
7016 <<lttng-ust-agents,LTTng-UST Python agent>>::
7017 A Python package provided by the LTTng-UST project to allow the
7018 LTTng instrumentation of Python logging statements.
7020 <<channel-overwrite-mode-vs-discard-mode,overwrite mode>>::
7021 The event loss mode in which new event records overwrite older
7022 event records when there's no sub-buffer space left to store them.
7024 <<channel-buffering-schemes,per-process buffering>>::
7025 A buffering scheme in which each instrumented process has its own
7026 sub-buffers for a given user space channel.
7028 <<channel-buffering-schemes,per-user buffering>>::
7029 A buffering scheme in which all the processes of a Unix user share the
7030 same sub-buffer for a given user space channel.
7032 <<lttng-relayd,relay daemon>>::
7033 A process which is responsible for receiving the trace data sent by
7034 a distant consumer daemon.
7037 A set of sub-buffers.
7039 <<lttng-sessiond,session daemon>>::
7040 A process which receives control commands from you and orchestrates
7041 the tracers and various LTTng daemons.
7043 <<taking-a-snapshot,snapshot>>::
7044 A copy of the current data of all the sub-buffers of a given tracing
7045 session, saved as trace files.
7048 One part of an LTTng ring buffer which contains event records.
7051 The time information attached to an event when it is emitted.
7054 A set of files which are the concatenations of one or more
7055 flushed sub-buffers.
7058 The action of recording the events emitted by an application
7059 or by a system, or to initiate such recording by controlling
7063 The http://tracecompass.org[Trace Compass] project and application.
7066 An instrumentation point using the tracepoint mechanism of the Linux
7067 kernel or of LTTng-UST.
7069 tracepoint definition::
7070 The definition of a single tracepoint.
7073 The name of a tracepoint.
7075 tracepoint provider::
7076 A set of functions providing tracepoints to an instrumented user
7079 Not to be confused with a _tracepoint provider package_: many tracepoint
7080 providers can exist within a tracepoint provider package.
7082 tracepoint provider package::
7083 One or more tracepoint providers compiled as an object file or as
7087 A software which records emitted events.
7089 <<domain,tracing domain>>::
7090 A namespace for event sources.
7092 <<tracing-group,tracing group>>::
7093 The Unix group in which a Unix user can be to be allowed to trace the
7096 <<tracing-session,tracing session>>::
7097 A stateful dialogue between you and a <<lttng-sessiond,session
7101 An application running in user space, as opposed to a Linux kernel
7102 module, for example.