1 .TH "LTTNG" "1" "July 17, 2012" "" ""
4 lttng \(em LTTng 2.0 tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which allows you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 We introduce the notion of \fBtracing domains\fP which is essentially a type of
32 tracer (kernel or user space for now). In the future, we could see a third
33 tracer being for instance an hypervisor. For some commands, you'll need to
34 specify on which domain the command applies (-u or -k). For instance, enabling
35 a kernel event, you must specify the kernel domain to the command so we know
36 for which tracer this event is for.
38 In order to trace the kernel, the session daemon needs to be running as root.
39 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
40 in that group can interact with the root session daemon and thus trace the
41 kernel. Session daemons can co-exist meaning that you can have a session daemon
42 running as Alice that can be used to trace her applications along side with a
43 root daemon or even a Bob daemon. We highly recommend to start the session
44 daemon at boot time for stable and long term tracing.
46 Every user-space applications instrumented with lttng-ust(3), will
47 automatically register to the session daemon. This feature gives you the
48 ability to list available traceable applications and tracepoints on a per user
49 basis. (See \fBlist\fP command).
53 This program follow the usual GNU command line syntax with long options starting with
54 two dashes. Below is a summary of the available options.
59 Show summary of possible options and commands.
61 .BR "\-v, \-\-verbose"
63 Three levels of verbosity are available which are triggered by putting additional v to
64 the option (\-vv or \-vvv)
67 Suppress all messages (even errors).
69 .BR "\-g, \-\-group NAME"
70 Set unix tracing group name. (default: tracing)
72 .BR "\-n, \-\-no-sessiond"
73 Don't automatically spawn a session daemon.
75 .BR "\-\-sessiond\-path"
76 Set session daemon full binary path.
78 .BR "\-\-list\-options"
79 Simple listing of lttng options.
81 .BR "\-\-list\-commands"
82 Simple listing of lttng commands.
88 Add context to event(s) and/or channel(s).
90 A context is basically extra information appended to a channel or event. For
91 instance, you could ask the tracer to add the PID information within the
92 "sched_switch" kernel event. You can also add performance monitoring unit
93 counters (perf PMU) using the perf kernel API).
95 For example, this command will add the context information 'prio' and two perf
96 counters (hardware branch misses and cache misses), to all events in the trace
99 # lttng add-context \-k \-t prio \-t perf:branch-misses \-t perf:cache-misses
101 Please take a look at the help (\-h/\-\-help) for a detailed list of available
104 If no channel and no event is given (\-c/\-e), the context is added to all
105 channels (which applies automatically to all events in that channel). Otherwise
106 the context will be added only to the channel (\-c) and/or event (\-e) indicated.
108 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
116 Show summary of possible options and commands.
117 \-s, \-\-session NAME
118 Apply on session name.
119 \-c, \-\-channel NAME
120 Apply on channel name.
124 Apply for the kernel tracer
126 Apply for the user-space tracer
128 Context type. You can repeat this option on the command line. Please
129 use "lttng add-context \-h" to list all available types.
134 .IP "\fBcalibrate\fP"
136 Quantify LTTng overhead
138 The LTTng calibrate command can be used to find out the combined average
139 overhead of the LTTng tracer and the instrumentation mechanisms used. This
140 overhead can be calibrated in terms of time or using any of the PMU performance
141 counter available on the system.
143 For now, the only calibration implemented is that of the kernel function
144 instrumentation (kretprobes).
146 * Calibrate kernel function instrumentation
148 Let's use an example to show this calibration. We use an i7 processor with 4
149 general-purpose PMU registers. This information is available by issuing dmesg,
150 looking for "generic registers".
152 This sequence of commands will gather a trace executing a kretprobe hooked on
153 an empty function, gathering PMU counters LLC (Last Level Cache) misses
154 information (see lttng add-context \-\-help to see the list of available PMU
157 # lttng create calibrate-function
158 # lttng enable-event calibrate \-\-kernel \-\-function lttng_calibrate_kretprobe
159 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \-t perf:LLC-store-misses \\
160 \-t perf:LLC-prefetch-misses
162 # for a in $(seq 1 10); do \\
163 lttng calibrate \-\-kernel \-\-function;
166 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* | tail \-n 1)
168 The output from babeltrace can be saved to a text file and opened in a
169 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
170 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
171 counters are per-CPU, so scheduling events would need to be present to account
172 for migration between CPU. Therefore, for calibration purposes, only events
173 staying on the same CPU must be considered.
175 The average result, for the i7, on 10 samples:
178 perf_LLC_load_misses: 5.0 0.577
179 perf_LLC_store_misses: 1.6 0.516
180 perf_LLC_prefetch_misses: 9.0 14.742
182 As we can notice, the load and store misses are relatively stable across runs
183 (their standard deviation is relatively low) compared to the prefetch misses.
184 We can conclude from this information that LLC load and store misses can be
185 accounted for quite precisely, but prefetches within a function seems to behave
186 too erratically (not much causality link between the code executed and the CPU
187 prefetch activity) to be accounted for.
194 Show summary of possible options and commands.
196 Apply for the kernel tracer
198 Apply for the user-space tracer
200 Dynamic function entry/return probe (default)
205 .IP "\fBcreate\fP [NAME] [OPTIONS]
207 Create tracing session.
209 A tracing session contains channel(s) which contains event(s). It is domain
210 agnostic meaning that you can enable channels and events for either the
211 user-space tracer and/or the kernel tracer. It acts like a container
212 aggregating multiple tracing sources.
214 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
215 containing the current session name. If NAME is omitted, a session name is
216 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
218 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
226 Show summary of possible options and commands.
228 Simple listing of options
230 Specify output path for traces
232 Using these options, each API call can be controlled individually. For
233 instance, \-C does not enable the consumer automatically. You'll need the \-e
237 Set URL for the enable-consumer destination. It is persistent for the
238 session lifetime. Redo the command to change it. This will set both
239 data and control URL for network.
240 \-C, \-\-ctrl-url=URL
241 Set control path URL. (Must use -D also)
242 \-D, \-\-data-url=URL
243 Set data path URL. (Must use -C also)
245 Don't activate a consumer for this session.
247 Disable consumer for this session.
249 See \fBenable-consumer\fP command below for the supported URL format.
253 # lttng create -U net://192.168.1.42
254 Uses TCP and default ports for the given destination.
256 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
257 Uses TCP, default ports and IPv6.
259 # lttng create s1 -U net://myhost.com:3229
260 Create session s1 and set its consumer to myhost.com on port 3229 for control.
265 .IP "\fBdestroy\fP [OPTIONS] [NAME]"
267 Teardown tracing session
269 Free memory on the session daemon and tracer side. It's gone!
271 If NAME is omitted, the session name is taken from the .lttngrc file.
278 Show summary of possible options and commands.
282 Simple listing of options
287 .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
289 Enable tracing channel
291 To enable event, you must first enable a channel which contains event(s).
293 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
303 Simple listing of options
305 Apply on session name
307 Apply to the kernel tracer
309 Apply to the user-space tracer
312 Discard event when subbuffers are full (default)
314 Flight recorder mode : overwrites events when subbuffers are full
316 Subbuffer size in bytes (default: 4096, kernel default: 262144)
318 Number of subbuffers (default: 4)
319 Needs to be a power of 2 for kernel and ust tracers
321 Switch subbuffer timer interval in usec (default: 0)
322 Needs to be a power of 2 for kernel and ust tracers
324 Read timer interval in usec (default: 200)
329 .IP "\fBenable-consumer\fP [-u|-k] [URL] [OPTIONS]"
331 Enable a consumer for the tracing session and domain.
333 By default, every tracing session has a consumer attached to it using the local
334 filesystem as output. The trace is written in $HOME/lttng-traces. This command
335 allows the user to specify a specific URL after the session was created for a
336 specific domain. If no domain is specified, the consumer is applied on all
339 Without options, the behavior is to enable a consumer to the current URL. The
340 default URL is the local filesystem at the path of the session mentioned above.
342 The enable-consumer feature supports both local and network transport. You must
343 have a running \fBlttng-relayd(8)\fP for network transmission or any other daemon
344 that can understand the streaming protocol of LTTng.
351 Show summary of possible options and commands.
353 Simple listing of options
355 Apply on session name
357 Apply for the kernel tracer
359 Apply for the user-space tracer
361 Using these options, each API call can be controlled individually. For
362 instance, \-C does not enable the consumer automatically. You'll need the \-e
366 Set URL for the enable-consumer destination. It is persistent for the
367 session lifetime. Redo the command to change it. This will set both
368 data and control URL for network.
369 \-C, \-\-ctrl-url=URL
370 Set control path URL. (Must use -D also)
371 \-D, \-\-data-url=URL
372 Set data path URL. (Must use -C also)
378 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
380 Supported protocols are (proto):
382 Local filesystem full path.
385 This will use the default network transport layer which is TCP for both
386 control (PORT1) and data port (PORT2). The default ports are
387 respectively 5342 and 5343.
390 Can only be used with -C and -D together
392 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
396 $ lttng enable-consumer -u net://192.168.1.42
398 Uses TCP and default ports for user space tracing (-u) where the IP address
399 above is the destination machine where the traces will be streamed and a
400 \fBlttng-relayd(8)\fP is listening.
403 .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
407 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
408 omitted, a default channel named '\fBchannel0\fP' is created and the event is
409 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
410 using the wildcard "*".
412 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
420 Show summary of possible options and commands.
422 Simple listing of options
424 Apply on session name
426 Apply on channel name
428 Enable all tracepoints and syscalls
430 Apply for the kernel tracer
432 Apply for the user-space tracer
435 Tracepoint event (default)
436 - userspace tracer supports wildcards at end of string. Don't forget to
437 quote to deal with bash expansion.
443 \-\-probe [addr | symbol | symbol+offset]
444 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
445 or hexadecimal (0xNNN...)
446 \-\-function [addr | symbol | symbol+offset]
447 Dynamic function entry/return probe. Addr and offset can be octal
448 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
450 System call event. Enabling syscalls tracing (kernel tracer), you will
451 not be able to disable them with disable-event. This is a known
452 limitation. You can disable the entire channel to do the trick.
454 \-\-filter 'expression'
455 Set a filter on a newly enabled event. Filter expression on event
456 fields, event recording depends on evaluation. Only specify on first
457 activation of a given event within a session. Filter only allowed when
458 enabling events within a session before tracing is started. If the
459 filter fails to link with the event within the traced domain, the event
460 will be discarded. Currently, filter is only implemented for the
465 'intfield > 500 && intfield < 503'
466 '(stringfield == "test" || intfield != 10) && intfield > 33'
467 'doublefield > 1.1 && intfield < 5.3'
469 Wildcards are allowed at the end of strings:
471 In string literals, the escape character is a '\\'. Use '\\*' for
472 the '*' character, and '\\\\' for the '\\' character.
475 .IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
477 Disable tracing channel
479 Disabling a channel makes all event(s) in that channel to stop tracing. You can
480 enable it back by calling \fBlttng enable-channel NAME\fP again.
482 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
490 Show summary of possible options and commands.
492 Simple listing of options
493 \-s, \-\-session NAME
494 Apply on session name
496 Apply for the kernel tracer
498 Apply for the user-space tracer
501 .IP "\fBdisable-consumer\fP [\-k|\-u] [OPTIONS]"
503 Disable the consumer of a tracing session.
505 This call MUST be done BEFORE tracing has started.
512 Show summary of possible options and commands.
514 Simple listing of options
515 \-s, \-\-session NAME
516 Apply on session name
518 Apply for the kernel tracer
520 Apply for the user-space tracer
523 .IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
525 Disable tracing event
527 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
530 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
538 Show summary of possible options and commands.
540 Simple listing of options
542 Apply on session name
544 Apply for the kernel tracer
546 Apply for the user-space tracer
549 .IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
551 List tracing session information.
553 With no arguments, it will list available tracing session(s).
555 With the session name, it will display the details of the session including
556 the trace file path, the associated channels and their state (activated
557 and deactivated), the activated events and more.
559 With \-k alone, it will list all available kernel events (except the system
561 With \-u alone, it will list all available user-space events from registered
562 applications. Here is an example of 'lttng list \-u':
564 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
565 ust_tests_hello:tptest_sighandler (type: tracepoint)
566 ust_tests_hello:tptest (type: tracepoint)
568 You can now enable any event listed by using the name :
569 \fBust_tests_hello:tptest\fP.
576 Show summary of possible options and commands.
578 Simple listing of options
582 Select user-space domain.
586 \-c, \-\-channel NAME
587 List details of a channel
589 List available domain(s)
592 .IP "\fBset-session\fP NAME"
594 Set current session name
596 Will change the session name in the .lttngrc file.
603 Show summary of possible options and commands.
605 Simple listing of options
610 .IP "\fBstart\fP [NAME] [OPTIONS]"
614 It will start tracing for all tracers for a specific tracing session.
616 If NAME is omitted, the session name is taken from the .lttngrc file.
623 Show summary of possible options and commands.
625 Simple listing of options
630 .IP "\fBstop\fP [NAME] [OPTIONS]"
634 It will stop tracing for all tracers for a specific tracing session.
636 If NAME is omitted, the session name is taken from the .lttngrc file.
643 Show summary of possible options and commands.
645 Simple listing of options
652 Show version information
659 Show summary of possible options and commands.
661 Simple listing of options
666 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
668 View traces of a tracing session
670 By default, the babeltrace viewer will be used for text viewing.
672 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
682 Simple listing of options
683 \-t, \-\-trace-path PATH
684 Trace directory path for the viewer
686 Specify viewer and/or options to use
687 This will completely override the default viewers so
688 please make sure to specify the full command. The trace
689 directory path of the session will be appended at the end
694 On success 0 is returned and a positive value on error. Value of 1 means a command
695 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
696 something went wrong during the command.
698 Any other value above 10, please refer to
699 .BR <lttng/lttng-error.h>
700 for a detailed list or use lttng_strerror() to get a human readable string of
704 .SH "ENVIRONMENT VARIABLES"
707 Note that all command line options override environment variables.
711 .IP "LTTNG_SESSIOND_PATH"
712 Allows one to specify the full session daemon binary path to lttng command line
713 tool. You can also use \-\-sessiond-path option having the same effect.
717 .BR lttng-sessiond(8),
719 .BR lttng-health-check(3)
722 With version 2.1 and earlier, if you start a tracing session and than enable
723 kernel events, they are not recorded and the tracing session fails to stop. To
724 fix this, simply enable events before starting the session.
726 If you encounter any issues or usability problem, please report it on our
727 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
728 at https://bugs.lttng.org which is a bugtracker.
732 lttng is distributed under the GNU General Public License version 2. See the file
735 A Web site is available at http://lttng.org for more information on the LTTng
738 You can also find our git tree at http://git.lttng.org.
740 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
742 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
747 Thanks to Yannick Brosseau without whom this project would never have been so
748 lean and mean! Also thanks to the Ericsson teams working on tracing which
749 helped us greatly with detailed bug reports and unusual test cases.
751 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
752 maintainer) and Jon Bernard for our Debian packages.
754 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
755 Montreal for the LTTng journey.
760 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
761 David Goulet. More people have since contributed to it. It is currently
762 maintained by David Goulet <dgoulet@efficios.com>.