1 .TH "LTTNG" "1" "December 3rd, 2012" "" ""
4 lttng \(em LTTng 2.x tracer control command line tool
10 lttng [OPTIONS] <COMMAND>
15 The LTTng project aims at providing highly efficient tracing tools for Linux.
16 It's tracers help tracking down performance issues and debugging problems
17 involving multiple concurrent processes and threads. Tracing across multiple
18 systems is also possible.
20 The \fBlttng\fP command line tool from the lttng-tools package is used to control
21 both kernel and user-space tracing. Every interactions with the tracer should
22 be done by this tool or by the liblttng-ctl provided with the lttng-tools
25 LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry,
26 which allows you to interact with multiple tracers (kernel and user-space)
27 inside the same container, a tracing session. Traces can be gathered from the
28 kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading
29 those traces is done using the babeltrace(1) text viewer.
31 We introduce the notion of \fBtracing domains\fP which is essentially a type of
32 tracer (kernel or user space for now). In the future, we could see a third
33 tracer being for instance an hypervisor. For some commands, you'll need to
34 specify on which domain the command applies (-u or -k). For instance, enabling
35 a kernel event, you must specify the kernel domain to the command so we know
36 for which tracer this event is for.
38 In order to trace the kernel, the session daemon needs to be running as root.
39 LTTng provides the use of a \fBtracing group\fP (default: tracing). Whomever is
40 in that group can interact with the root session daemon and thus trace the
41 kernel. Session daemons can co-exist meaning that you can have a session daemon
42 running as Alice that can be used to trace her applications along side with a
43 root daemon or even a Bob daemon. We highly recommend to start the session
44 daemon at boot time for stable and long term tracing.
46 Every user-space applications instrumented with lttng-ust(3), will
47 automatically register to the session daemon. This feature gives you the
48 ability to list available traceable applications and tracepoints on a per user
49 basis. (See \fBlist\fP command).
53 This program follow the usual GNU command line syntax with long options starting with
54 two dashes. Below is a summary of the available options.
59 Show summary of possible options and commands.
61 .BR "\-v, \-\-verbose"
63 Three levels of verbosity are available which are triggered by putting additional v to
64 the option (\-vv or \-vvv)
67 Suppress all messages (even errors).
69 .BR "\-g, \-\-group NAME"
70 Set unix tracing group name. (default: tracing)
72 .BR "\-n, \-\-no-sessiond"
73 Don't automatically spawn a session daemon.
75 .BR "\-\-sessiond\-path PATH"
76 Set session daemon full binary path.
78 .BR "\-\-list\-options"
79 Simple listing of lttng options.
81 .BR "\-\-list\-commands"
82 Simple listing of lttng commands.
88 Add context to event(s) and/or channel(s).
90 A context is basically extra information appended to a channel. For instance,
91 you could ask the tracer to add the PID information for all events in a
92 channel. You can also add performance monitoring unit counters (perf PMU) using
95 For example, this command will add the context information 'prio' and two perf
96 counters (hardware branch misses and cache misses), to all events in the trace
100 # lttng add-context \-k \-t prio \-t perf:branch-misses \\
101 \-t perf:cache-misses
104 Please take a look at the help (\-h/\-\-help) for a detailed list of available
107 If no channel is given (\-c), the context is added to all channels that were
108 already enabled. If the session has no channel, a default channel is created.
109 Otherwise the context will be added only to the given channel (\-c).
111 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
118 Show summary of possible options and commands.
120 .BR "\-s, \-\-session NAME"
121 Apply on session name.
123 .BR "\-c, \-\-channel NAME"
124 Apply on channel name.
126 .BR "\-k, \-\-kernel"
127 Apply for the kernel tracer
129 .BR "\-u, \-\-userspace"
130 Apply for the user-space tracer
132 .BR "\-t, \-\-type TYPE"
133 Context type. You can repeat this option on the command line. Please
134 use "lttng add-context \-h" to list all available types.
140 .IP "\fBcalibrate\fP"
142 Quantify LTTng overhead
144 The LTTng calibrate command can be used to find out the combined average
145 overhead of the LTTng tracer and the instrumentation mechanisms used. This
146 overhead can be calibrated in terms of time or using any of the PMU performance
147 counter available on the system.
149 For now, the only calibration implemented is that of the kernel function
150 instrumentation (kretprobes).
152 * Calibrate kernel function instrumentation
154 Let's use an example to show this calibration. We use an i7 processor with 4
155 general-purpose PMU registers. This information is available by issuing dmesg,
156 looking for "generic registers".
158 This sequence of commands will gather a trace executing a kretprobe hooked on
159 an empty function, gathering PMU counters LLC (Last Level Cache) misses
160 information (see lttng add-context \-\-help to see the list of available PMU
163 # lttng create calibrate-function
164 # lttng enable-event calibrate \-\-kernel \-\-function lttng_calibrate_kretprobe
165 # lttng add-context \-\-kernel \-t perf:LLC-load-misses \-t perf:LLC-store-misses \\
166 \-t perf:LLC-prefetch-misses
168 # for a in $(seq 1 10); do \\
169 lttng calibrate \-\-kernel \-\-function;
172 # babeltrace $(ls \-1drt ~/lttng-traces/calibrate-function-* | tail \-n 1)
174 The output from babeltrace can be saved to a text file and opened in a
175 spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
176 consecutive "calibrate_entry" and "calibrate_return" events. Note that these
177 counters are per-CPU, so scheduling events would need to be present to account
178 for migration between CPU. Therefore, for calibration purposes, only events
179 staying on the same CPU must be considered.
181 The average result, for the i7, on 10 samples:
184 perf_LLC_load_misses: 5.0 0.577
185 perf_LLC_store_misses: 1.6 0.516
186 perf_LLC_prefetch_misses: 9.0 14.742
188 As we can notice, the load and store misses are relatively stable across runs
189 (their standard deviation is relatively low) compared to the prefetch misses.
190 We can conclude from this information that LLC load and store misses can be
191 accounted for quite precisely, but prefetches within a function seems to behave
192 too erratically (not much causality link between the code executed and the CPU
193 prefetch activity) to be accounted for.
200 Show summary of possible options and commands.
202 Apply for the kernel tracer
204 Apply for the user-space tracer
206 Dynamic function entry/return probe (default)
211 .IP "\fBcreate\fP [NAME] [OPTIONS]
213 Create tracing session.
215 A tracing session contains channel(s) which contains event(s). It is domain
216 agnostic meaning that you can enable channels and events for either the
217 user-space tracer and/or the kernel tracer. It acts like a container
218 aggregating multiple tracing sources.
220 On creation, a \fB.lttngrc\fP file is created in your $HOME directory
221 containing the current session name. If NAME is omitted, a session name is
222 automatically created having this form: 'auto-yyyymmdd-hhmmss'.
224 If no \fB\-o, \-\-output\fP is specified, the traces will be written in
227 The $HOME environment variable can be overridden by defining the environment
228 variable LTTNG_HOME. This is useful when the user running the commands has
229 a non-writeable home directory.
236 Show summary of possible options and commands.
238 Simple listing of options
240 Specify output path for traces
242 Traces will not be outputed
244 Set the session in snapshot mode. Created in no-output mode
245 and uses the URL, if one, as the default snapshot output.
246 Every channel will be set in overwrite mode and with mmap
247 output (splice not supported).
249 Using these options, each API call can be controlled individually. For
250 instance, \-C does not enable the consumer automatically. You'll need the \-e
254 Set URL for the consumer output destination. It is persistent for the
255 session lifetime. Redo the command to change it. This will set both
256 data and control URL for network.
257 \-C, \-\-ctrl-url=URL
258 Set control path URL. (Must use -D also)
259 \-D, \-\-data-url=URL
260 Set data path URL. (Must use -C also)
264 proto://[HOST|IP][:PORT1[:PORT2]][/TRACE_PATH]
266 Supported protocols are (proto):
268 Local filesystem full path.
271 This will use the default network transport layer which is TCP for both
272 control (PORT1) and data port (PORT2). The default ports are
273 respectively 5342 and 5343. Note that net[6]:// is not yet supported.
276 Can only be used with -C and -D together
278 NOTE: IPv6 address MUST be enclosed in brackets '[]' (rfc2732)
282 # lttng create -U net://192.168.1.42
283 Uses TCP and default ports for the given destination.
285 # lttng create -U net6://[fe80::f66d:4ff:fe53:d220]
286 Uses TCP, default ports and IPv6.
288 # lttng create s1 -U net://myhost.com:3229
289 Create session s1 and set its consumer to myhost.com on port 3229 for control.
294 .IP "\fBdestroy\fP [OPTIONS] [NAME]"
296 Teardown tracing session
298 Free memory on the session daemon and tracer side. It's gone!
300 If NAME is omitted, the session name is taken from the .lttngrc file.
307 Show summary of possible options and commands.
311 Simple listing of options
316 .IP "\fBenable-channel\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
318 Enable tracing channel
320 To enable an event, you must enable both the event and the channel that
323 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
326 It is important to note that if a certain type of buffers is used, the session
327 will be set with that type and all other subsequent channel needs to have the
330 Note that once the session has been started and enabled on the tracer side,
331 it's not possible anymore to enable a new channel for that session.
340 Simple listing of options
341 \-s, \-\-session NAME
342 Apply on session name
344 Apply to the kernel tracer
346 Apply to the user-space tracer
349 Discard event when subbuffers are full (default)
351 Flight recorder mode : overwrites events when subbuffers are full
353 Subbuffer size in bytes {+k,+M,+G}
354 (default UST uid: 131072, UST pid: 4096, kernel: 262144, metadata: 4096)
355 Rounded up to the next power of 2.
357 The minimum subbuffer size, for each tracer, is the max value between
358 the default above and the system page size. You can issue this command
359 to get the current page size on your system: \fB$ getconf PAGE_SIZE\fP
361 Number of subbuffers. (default UST uid: 4, UST pid: 4, kernel: 4, metadata: 2)
362 Rounded up to the next power of 2.
363 \-\-switch-timer USEC
364 Switch subbuffer timer interval in µsec.
365 (default UST uid: 0, UST pid: 0, kernel: 0, metadata: 0)
367 Read timer interval in µsec.
368 (default UST uid: 0, UST pid: 0, kernel: 200000, metadata: 0)
370 Channel output type. Possible values: mmap, splice
371 (default UST uid: mmap, UST pid: mmap, kernel: splice, metadata: mmap)
373 Use per UID buffer (\-u only). Buffers are shared between applications
374 that have the same UID.
376 Use per PID buffer (\-u only). Each application has its own buffers.
378 Use shared buffer for the whole system (\-k only)
379 \-C, \-\-tracefile-size SIZE
380 Maximum size of each tracefile within a stream (in bytes).
381 0 means unlimited. (default: 0)
382 \-W, \-\-tracefile-count COUNT
383 Used in conjunction with \-C option, this will limit the number
384 of files created to the specified count. 0 means unlimited. (default: 0)
388 $ lttng enable-channel -C 4096 -W 32 chan1
389 For each stream, the maximum size of each trace file will be 4096 bytes, and
390 there will be a maximum of 32 different files. The file count is appended after
391 the stream number as seen in the following example. The last trace file is
392 smaller than 4096 since it was not completely filled.
394 ~/lttng-traces/[...]/chan1_0_0 (4096)
395 ~/lttng-traces/[...]/chan1_0_1 (4096)
396 ~/lttng-traces/[...]/chan1_0_2 (3245)
397 ~/lttng-traces/[...]/chan1_1_0 (4096)
400 $ lttng enable-channel -C 4096
401 This will create trace files of 4096 bytes and will create new ones as long as
402 there is data available.
407 .IP "\fBenable-event\fP NAME[,NAME2,...] [-k|-u] [OPTIONS]"
411 A tracing event is always assigned to a channel. If \fB\-c, \-\-channel\fP is
412 omitted, a default channel named '\fBchannel0\fP' is created and the event is
413 added to it. For the user-space tracer, using \fB\-a, \-\-all\fP is the same as
414 using the wildcard "*".
416 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
424 Show summary of possible options and commands.
426 Simple listing of options
427 \-s, \-\-session NAME
428 Apply on session name
429 \-c, \-\-channel NAME
430 Apply on channel name
432 Enable all tracepoints and syscalls. This actually enable a single
435 Apply for the kernel tracer
437 Apply for the user-space tracer
440 Tracepoint event (default)
441 - userspace tracer supports wildcards at end of string. Don't forget to
442 quote to deal with bash expansion.
447 Tracepoint loglevel range from 0 to loglevel. Listed in the help (\-h).
448 \-\-loglevel-only NAME
449 Tracepoint loglevel (only this loglevel).
451 The loglevel or loglevel-only options should be combined with a
452 tracepoint name or tracepoint wildcard.
453 \-\-probe [addr | symbol | symbol+offset]
454 Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
455 or hexadecimal (0xNNN...)
456 \-\-function [addr | symbol | symbol+offset]
457 Dynamic function entry/return probe. Addr and offset can be octal
458 (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
460 System call event. Enabling syscalls tracing (kernel tracer), you will
461 not be able to disable them with disable-event. This is a known
462 limitation. You can disable the entire channel to do the trick.
464 \-\-filter 'expression'
465 Set a filter on a newly enabled event. Filter expression on event
466 fields and context. Event recording depends on evaluation. Only
467 specify on first activation of a given event within a session.
468 Filter only allowed when enabling events within a session before
469 tracing is started. If the filter fails to link with the event
470 within the traced domain, the event will be discarded.
471 Currently, filter is only implemented for the user-space tracer.
475 'intfield > 500 && intfield < 503'
476 '(stringfield == "test" || intfield != 10) && intfield > 33'
477 'doublefield > 1.1 && intfield < 5.3'
479 Wildcards are allowed at the end of strings:
481 In string literals, the escape character is a '\\'. Use '\\*' for
482 the '*' character, and '\\\\' for the '\\' character. Wildcard
483 match any sequence of characters, including an empty sub-string
484 (match 0 or more characters).
486 Context information can be used for filtering. The examples
487 below show usage of context filtering on process name (with a
488 wildcard), process ID range, and unique thread ID for filtering.
489 The process and thread ID of running applications can be found
490 under columns "PID" and "LWP" of the "ps -eLf" command.
492 '$ctx.procname == "demo*"'
493 '$ctx.vpid >= 4433 && $ctx.vpid < 4455'
497 .IP "\fBdisable-channel\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
499 Disable tracing channel
501 Disabling a channel makes all event(s) in that channel to stop tracing. You can
502 enable it back by calling \fBlttng enable-channel NAME\fP again.
504 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
512 Show summary of possible options and commands.
514 Simple listing of options
515 \-s, \-\-session NAME
516 Apply on session name
518 Apply for the kernel tracer
520 Apply for the user-space tracer
523 .IP "\fBdisable-event\fP NAME[,NAME2,...] [\-k|\-u] [OPTIONS]"
525 Disable tracing event
527 The event, once disabled, can be re-enabled by calling \fBlttng enable-event
530 If \fB\-s, \-\-session\fP is omitted, the session name is taken from the .lttngrc
538 Show summary of possible options and commands.
540 Simple listing of options
541 \-s, \-\-session NAME
542 Apply on session name
544 Disable all events. This does NOT disable "*" but rather
545 every known events of the session.
547 Apply for the kernel tracer
549 Apply for the user-space tracer
552 .IP "\fBlist\fP [\-k|\-u] [SESSION [SESSION_OPTIONS]]"
554 List tracing session information.
556 With no arguments, it will list available tracing session(s).
558 With the session name, it will display the details of the session including
559 the trace file path, the associated channels and their state (activated
560 and deactivated), the activated events and more.
562 With \-k alone, it will list all available kernel events (except the system
564 With \-u alone, it will list all available user-space events from registered
565 applications. Here is an example of 'lttng list \-u':
567 PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
568 ust_tests_hello:tptest_sighandler (type: tracepoint)
569 ust_tests_hello:tptest (type: tracepoint)
571 You can now enable any event listed by using the name :
572 \fBust_tests_hello:tptest\fP.
579 Show summary of possible options and commands.
581 Simple listing of options
585 Select user-space domain.
589 \-c, \-\-channel NAME
590 List details of a channel
592 List available domain(s)
595 .IP "\fBset-session\fP NAME"
597 Set current session name
599 Will change the session name in the .lttngrc file.
606 Show summary of possible options and commands.
608 Simple listing of options
613 .IP "\fBsnapshot\fP ACTION"
615 Snapshot command for LTTng session.
622 Show summary of possible options and commands.
624 Simple listing of options
630 \fBadd-output\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] <URL> | -C <URL> -D <URL>
632 Setup and add an snapshot output for a session. Output are the destination
633 where the snapshot will be sent. Only one output is permitted. To change it,
634 you'll need to delete it and add back the new one.
636 \fBdel-output\fP ID | NAME [-s <NAME>]
638 Delete an output for a session using the ID. You can either specify the
639 output's ID that can be found with list-output or the name.
641 \fBlist-output\fP [-s <NAME>]
643 List the output of a session. Attributes of the output are printed.
645 \fBrecord\fP [-m <SIZE>] [-s <NAME>] [-n <NAME>] [<URL> | -C <URL> -D <URL>]
647 Snapshot a session's buffer(s) for all domains. If an URL is specified, it is
648 used instead of a previously added output. Specifying only a name or/and a max
649 size will override the current output values. For instance, you can record a
650 snapshot with a custom maximum size or with a different name.
652 $ lttng add-output -n mysnapshot file:///data/snapshot
654 $ lttng snapshot record -n new_name_snapshot
656 The above will create a snapshot in /data/snapshot/new_name_snapshot* directory
657 rather then in mysnapshot*/
663 \-s, \-\-session NAME
664 Apply to session name.
666 Name of the snapshot's output.
667 \-m, \-\-max-size SIZE
668 Maximum size in bytes of the snapshot. The maxium size does not
669 include the metadata file.
670 \-C, \-\-ctrl-url URL
671 Set control path URL. (Must use -D also)
672 \-D, \-\-data-url URL
673 Set data path URL. (Must use -C also)
678 .IP "\fBstart\fP [NAME] [OPTIONS]"
682 It will start tracing for all tracers for a specific tracing session.
684 If NAME is omitted, the session name is taken from the .lttngrc file.
691 Show summary of possible options and commands.
693 Simple listing of options
698 .IP "\fBstop\fP [NAME] [OPTIONS]"
702 It will stop tracing for all tracers for a specific tracing session. Before
703 returning, the command checks for data availability meaning that it will wait
704 until the trace is readable for the session. Use \-\-no-wait to avoid this
707 If NAME is omitted, the session name is taken from the .lttngrc file.
714 Show summary of possible options and commands.
716 Simple listing of options
718 Don't wait for data availability.
725 Show version information
732 Show summary of possible options and commands.
734 Simple listing of options
739 .IP "\fBview\fP [SESSION_NAME] [OPTIONS]"
741 View traces of a tracing session
743 By default, the babeltrace viewer will be used for text viewing.
745 If SESSION_NAME is omitted, the session name is taken from the .lttngrc file.
755 Simple listing of options
756 \-t, \-\-trace-path PATH
757 Trace directory path for the viewer
759 Specify viewer and/or options to use
760 This will completely override the default viewers so
761 please make sure to specify the full command. The trace
762 directory path of the session will be appended at the end
767 On success 0 is returned and a positive value on error. Value of 1 means a command
768 error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that
769 something went wrong during the command.
771 Any other value above 10, please refer to
772 .BR <lttng/lttng-error.h>
773 for a detailed list or use lttng_strerror() to get a human readable string of
777 .SH "ENVIRONMENT VARIABLES"
780 Note that all command line options override environment variables.
784 .IP "LTTNG_SESSIOND_PATH"
785 Allows one to specify the full session daemon binary path to lttng command line
786 tool. You can also use \-\-sessiond-path option having the same effect.
790 .BR lttng-sessiond(8),
792 .BR lttng-health-check(3)
795 If you encounter any issues or usability problem, please report it on our
796 mailing list <lttng-dev@lists.lttng.org> to help improve this project or
797 at https://bugs.lttng.org which is a bugtracker.
801 lttng is distributed under the GNU General Public License version 2. See the file
804 A Web site is available at http://lttng.org for more information on the LTTng
807 You can also find our git tree at http://git.lttng.org.
809 Mailing lists for support and development: <lttng-dev@lists.lttng.org>.
811 You can find us on IRC server irc.oftc.net (OFTC) in #lttng.
816 Thanks to Yannick Brosseau without whom this project would never have been so
817 lean and mean! Also thanks to the Ericsson teams working on tracing which
818 helped us greatly with detailed bug reports and unusual test cases.
820 Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA
821 maintainer) and Jon Bernard for our Debian packages.
823 Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de
824 Montreal for the LTTng journey.
829 lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and
830 David Goulet. More people have since contributed to it. It is currently
831 maintained by David Goulet <dgoulet@efficios.com>.