Commit | Line | Data |
---|---|---|
5cd1099a PMF |
1 | \input texinfo @c -*-texinfo-*- |
2 | @c %**start of header | |
3 | @setfilename ust.info | |
4 | @settitle LTTng Userspace Tracer (UST) Manual | |
5 | @c %**end of header | |
6 | ||
7 | @copying | |
8 | This manual is for program, version version. | |
9 | ||
10 | Copyright @copyright{} copyright-owner. | |
11 | ||
12 | @quotation | |
13 | Permission is granted to ... | |
14 | @end quotation | |
15 | @end copying | |
16 | ||
17 | @titlepage | |
18 | @title LTTng Userspace Tracer (UST) Manual | |
19 | @c @subtitle subtitle-if-any | |
20 | @c @subtitle second-subtitle | |
21 | @c @author author | |
22 | ||
23 | @c The following two commands | |
24 | @c start the copyright page. | |
25 | @c @page | |
26 | @c @vskip 0pt plus 1filll | |
27 | @c @insertcopying | |
28 | ||
29 | @c Published by ... | |
30 | @end titlepage | |
31 | ||
32 | @c So the toc is printed at the start. | |
33 | @contents | |
34 | ||
35 | @ifnottex | |
36 | @node Top | |
37 | @top LTTng Userspace Tracer | |
38 | ||
df39c97f | 39 | This manual is for UST 0.11. |
5cd1099a PMF |
40 | @end ifnottex |
41 | ||
42 | @menu | |
43 | * Overview:: | |
44 | * Installation:: | |
45 | * Quick start:: | |
46 | * Instrumenting an application:: | |
47 | * Recording a trace:: | |
48 | * Viewing traces:: | |
49 | * Performance:: | |
bb0496b2 | 50 | * Resource Usage:: |
4bea9b39 PMF |
51 | * List of environment variables detected by libust:: |
52 | * GDB integration:: | |
5cd1099a PMF |
53 | @c * Copying:: Your rights and freedoms. |
54 | @end menu | |
55 | ||
56 | @node Overview | |
57 | @chapter Overview | |
58 | ||
59 | @menu | |
70e55dbe | 60 | * What is UST?:: |
79f9fac7 | 61 | * License:: |
5cd1099a PMF |
62 | * Supported platforms:: |
63 | @end menu | |
64 | ||
70e55dbe PMF |
65 | @node What is UST? |
66 | @section What is UST? | |
67 | ||
68 | The LTTng Userspace Tracer (UST) is a library accompanied by a set of tools to | |
69 | trace userspace code. | |
70 | ||
71 | Code may be instrumented with either markers or tracepoints. A highly efficient | |
72 | lockless tracer records these events to a trace buffers. These buffers are reaped | |
73 | by a deamon which writes trace data to disk. | |
74 | ||
75 | High performance is achieved by the use of lockless buffering algorithms, RCU and | |
76 | per-cpu buffers. In addition, special care is taken to minize cache impact. | |
77 | ||
79f9fac7 PMF |
78 | @node License |
79 | @section License | |
80 | The LTTng Userspace Tracer is intended to be linkable to open source software | |
81 | as well as to proprietary applications. This was accomplished by licensing | |
82 | the code that needs to be linked to the traced program as @acronym{LGPL}. | |
83 | ||
675ac4ab PMF |
84 | Components licensed as LGPL v2.1: |
85 | @itemize @bullet | |
86 | @item libust | |
87 | @item libinterfork | |
88 | @item libustcomm | |
89 | @end itemize | |
90 | ||
91 | Components licensed as GPL v2: | |
92 | @itemize @bullet | |
93 | @item ustctl | |
2298f329 | 94 | @item libustctl |
9dc7b7ff | 95 | @item ust-consumerd |
675ac4ab | 96 | @end itemize |
79f9fac7 | 97 | |
5cd1099a PMF |
98 | @node Supported platforms |
99 | @section Supported platforms | |
100 | ||
bb0496b2 PMF |
101 | UST can currently trace applications running on Linux, on the x86-32, x86-64 |
102 | and PowerPC 32 architectures. | |
5cd1099a PMF |
103 | |
104 | @node Installation | |
105 | @chapter Installation | |
106 | ||
107 | The LTTng userspace tracer is a library and a set of userspace tools. | |
108 | ||
109 | The following packages are required: | |
110 | ||
111 | @itemize @bullet | |
112 | @item | |
113 | ust | |
114 | ||
9dc7b7ff | 115 | This contains the tracing library, the ust-consumerd daemon, trace control tools |
5cd1099a PMF |
116 | and other helper tools. |
117 | ||
72d022d6 | 118 | Repository: @url{http://git.dorsal.polymtl.ca} |
5cd1099a PMF |
119 | |
120 | @item | |
121 | liburcu | |
122 | ||
123 | This is the userspace read-copy update library by Mathieu Desnoyers. | |
124 | ||
125 | Available in Debian as package liburcu-dev. | |
126 | ||
72d022d6 | 127 | Home page: @url{http://lttng.org/urcu} |
5cd1099a PMF |
128 | |
129 | @item | |
130 | LTTV | |
131 | ||
132 | LTTV is a graphical (and text) viewer for LTTng traces. | |
133 | ||
72d022d6 | 134 | Home page: @url{http://lttng.org} |
5cd1099a PMF |
135 | |
136 | @end itemize | |
137 | ||
72d022d6 PMF |
138 | Liburcu should be installed first. UST may then be compiled and installed. LTTV |
139 | has no dependency on the other packages; it may therefore be installed on a | |
140 | system which does not have UST installed. | |
5cd1099a PMF |
141 | |
142 | Refer to the README in each of these packages for installation instructions. | |
143 | ||
144 | @c @menu | |
145 | @c @end menu | |
146 | ||
147 | @node Quick start | |
148 | @chapter Quick start | |
149 | ||
150 | First, instrument a program with a marker. | |
151 | ||
152 | @example | |
153 | @verbatim | |
154 | ||
155 | #include <ust/marker.h> | |
156 | ||
157 | int main(int argc, char **argv) | |
158 | { | |
159 | int v; | |
160 | char *st; | |
161 | ||
162 | /* ... set values of v and st ... */ | |
163 | ||
164 | /* a marker: */ | |
e367f87d | 165 | ust_marker(myevent, "firstarg %d secondarg %s", v, st); |
5cd1099a PMF |
166 | |
167 | /* a marker without arguments: */ | |
e367f87d | 168 | ust_marker(myotherevent, MARK_NOARGS); |
5cd1099a PMF |
169 | |
170 | return 0; | |
171 | } | |
172 | ||
173 | @end verbatim | |
174 | @end example | |
175 | ||
176 | Then compile it in the regular way, linking it with libust. For example: | |
177 | ||
178 | @example | |
179 | gcc -o foo -lust foo.c | |
180 | @end example | |
181 | ||
182 | Run the program with @command{usttrace}. The @command{usttrace} output says where the trace | |
183 | was written. | |
184 | ||
185 | @example | |
186 | usttrace ./foo | |
187 | @end example | |
188 | ||
189 | Finally, open the trace in LTTV. | |
190 | ||
191 | @example | |
192 | lttv-gui -t /path/to/trace | |
193 | @end example | |
194 | ||
195 | The trace can also be dumped as text in the console: | |
196 | ||
197 | @example | |
198 | lttv -m textDump -t /path/to/trace | |
199 | @end example | |
200 | ||
201 | @node Instrumenting an application | |
202 | @chapter Instrumenting an application | |
203 | ||
204 | In order to record a trace of events occurring in a application, the | |
205 | application must be instrumented. Instrumentation points resemble function | |
206 | calls. When the program reaches an instrumentation point, an event is | |
207 | generated. | |
208 | ||
209 | There are no limitations on the type of code that may be instrumented. | |
210 | Multi-threaded programs may be instrumented without problem. Signal handlers | |
211 | may be instrumented as well. | |
212 | ||
213 | There are two APIs to instrument programs: markers and tracepoints. Markers are | |
214 | quick to add and are usually used for temporary instrumentation. Tracepoints | |
215 | provide a way to instrument code more cleanly and are suited for permanent | |
216 | instrumentation. | |
217 | ||
218 | In addition to executable programs, shared libraries may also be instrumented | |
219 | with the methods described in this chapter. | |
220 | ||
221 | @menu | |
222 | * Markers:: | |
223 | * Tracepoints:: | |
224 | @end menu | |
225 | ||
226 | @node Markers | |
227 | @section Markers | |
228 | ||
70e55dbe | 229 | Adding a marker is simply a matter of inserting one line in the program. |
5cd1099a PMF |
230 | |
231 | @example | |
232 | @verbatim | |
233 | #include <ust/marker.h> | |
234 | ||
235 | int main(int argc, char **argv) | |
236 | { | |
237 | int v; | |
238 | char *st; | |
239 | ||
240 | /* ... set values of v and st ... */ | |
241 | ||
242 | /* a marker: */ | |
e367f87d | 243 | ust_marker(myevent, "firstarg %d secondarg %s", v, st); |
5cd1099a | 244 | |
70e55dbe | 245 | /* another marker without arguments: */ |
e367f87d | 246 | ust_marker(myotherevent, MARK_NOARGS); |
5cd1099a PMF |
247 | |
248 | return 0; | |
249 | } | |
250 | @end verbatim | |
251 | @end example | |
252 | ||
e367f87d MD |
253 | The invocation of the ust_marker() macro requires at least 2 arguments. The |
254 | first, "myevent", is the name of the event. The second is a format string | |
255 | that announces the names and the types of the event arguments. Its | |
256 | format resembles that of a printf() format string; it is described | |
257 | thoroughly in Appendix x. | |
5cd1099a PMF |
258 | |
259 | A given Marker may appear more than once in the same program. Other Markers may | |
260 | have the same name and a different format string, although this might induce | |
261 | some confusion at analysis time. | |
262 | ||
263 | @node Tracepoints | |
264 | @section Tracepoints | |
265 | ||
266 | The Tracepoints API uses the Markers, but provides a higher-level abstraction. | |
267 | Whereas the markers API provides limited type checking, the Tracepoints API | |
268 | provides more thorough type checking and discharges from the need to insert | |
269 | format strings directly in the code and to have format strings appear more than | |
270 | once if a given marker is reused. | |
271 | ||
bb0496b2 | 272 | @quotation Note |
e367f87d | 273 | The @command{usttrace} tool always uses the early tracing mode. |
bb0496b2 | 274 | @end quotation |
5cd1099a PMF |
275 | |
276 | A function instrumented with a tracepoint looks like this: | |
277 | ||
278 | @example | |
279 | @verbatim | |
280 | #include "tp.h" | |
281 | ||
e367f87d | 282 | void function(void) |
5cd1099a PMF |
283 | { |
284 | int v; | |
285 | char *st; | |
286 | ||
287 | /* ... set values of v and st ... */ | |
288 | ||
289 | /* a tracepoint: */ | |
e367f87d | 290 | trace_myevent(v, st); |
5cd1099a PMF |
291 | } |
292 | @end verbatim | |
293 | @end example | |
294 | ||
295 | Another file, here tp.h, contains declarations for the tracepoint. | |
296 | ||
297 | @example | |
298 | @verbatim | |
299 | #include <ust/tracepoint.h> | |
300 | ||
e367f87d MD |
301 | DECLARE_TRACEPOINT(myevent, TP_PROTO(int v, char *st), |
302 | TP_ARGS(v, st)); | |
5cd1099a PMF |
303 | @end verbatim |
304 | @end example | |
305 | ||
306 | A third file, here tp.c, contains definitions for the tracepoint. | |
307 | ||
308 | @example | |
309 | @verbatim | |
310 | #include <ust/marker.h> | |
311 | #include "tp.h" | |
312 | ||
e367f87d | 313 | DEFINE_TRACEPOINT(myevent); |
5cd1099a | 314 | |
e367f87d | 315 | void myevent_probe(int v, char *st) |
5cd1099a | 316 | { |
e367f87d | 317 | ust_marker(myevent, "v %d st %s", v, st); |
5cd1099a PMF |
318 | } |
319 | ||
320 | static void __attribute__((constructor)) init() | |
321 | { | |
e367f87d | 322 | register_trace_myevent(myevent_probe); |
5cd1099a PMF |
323 | } |
324 | @end verbatim | |
325 | @end example | |
326 | ||
327 | Here, tp.h and tp.c could contain declarations and definitions for other | |
328 | tracepoints. The constructor would contain other register_* calls. | |
329 | ||
330 | @node Recording a trace | |
331 | @chapter Recording a trace | |
332 | ||
333 | @menu | |
334 | * Using @command{usttrace}:: | |
335 | * Setting up the recording manually:: | |
336 | * Using early tracing:: | |
337 | * Crash recovery:: | |
338 | * Tracing across @code{fork()} and @code{clone()}:: | |
339 | * Tracing programs and libraries that were not linked to libust:: | |
340 | @end menu | |
341 | ||
342 | @node Using @command{usttrace} | |
343 | @section Using @command{usttrace} | |
344 | ||
345 | The simplest way to record a trace is to use the @command{usttrace} script. An | |
346 | example is given in the quickstart above. | |
347 | ||
348 | The @command{usttrace} script automatically: | |
349 | @itemize @bullet | |
350 | @item creates a daemon | |
351 | @item enables all markers | |
352 | @item runs the command specified on the command line | |
353 | @item after the command ends, prints the location where the trace was saved | |
354 | @end itemize | |
355 | ||
356 | Each subdirectory of the save location contains the trace of one process that | |
357 | was generated by the command. The name of a subdirectory consists in the the PID | |
358 | of the process, followed by the timestamp of its creation. | |
359 | ||
360 | The save location also contains logs of the tracing. | |
361 | ||
362 | When using @command{usttrace}, the early tracing is always active, which means | |
363 | that the tracing is guaranteed to be started by the time the process enters its | |
4bea9b39 | 364 | @code{main()} function. |
5cd1099a PMF |
365 | |
366 | Several @command{usttrace}'s may be run simultaneously without risk of | |
367 | conflict. This facilitates the use of the tracer by idependent users on a | |
368 | system. Each instance of @command{usttrace} starts its own daemon which | |
369 | collects the events of the processes it creates. | |
370 | ||
371 | @node Setting up the recording manually | |
372 | @section Setting up the recording manually | |
373 | ||
374 | Instead of using @command{usttrace}, a trace may be recorded on an already | |
375 | running process. | |
376 | ||
377 | First the daemon must be started. | |
378 | ||
379 | @example | |
380 | @verbatim | |
381 | # Make sure the directory for the communication sockets exists. | |
382 | $ mkdir /tmp/ustsocks | |
383 | ||
9dc7b7ff | 384 | # Make sure the directory where ust-consumerd will write the trace exists. |
5cd1099a PMF |
385 | $ mkdir /tmp/trace |
386 | ||
387 | # Start the daemon | |
9dc7b7ff | 388 | $ ust-consumerd |
5cd1099a PMF |
389 | |
390 | # We assume the program we want to trace is already running and that | |
391 | # it has pid 1234. | |
392 | ||
393 | # List the available markers | |
eb78cc5f | 394 | $ ustctl list-markers 1234 |
5cd1099a PMF |
395 | # A column indicates 0 for an inactive marker and 1 for an active marker. |
396 | ||
397 | # Enable a marker | |
eb78cc5f | 398 | $ ustctl enable-marker 1234 auto ust/mymark |
5cd1099a PMF |
399 | |
400 | # Create a trace | |
eb78cc5f | 401 | $ ustctl create-trace 1234 auto |
5cd1099a PMF |
402 | |
403 | # Start tracing | |
eb78cc5f | 404 | $ ustctl start-trace 1234 auto |
5cd1099a PMF |
405 | |
406 | # Do things... | |
407 | ||
408 | # Stop tracing | |
eb78cc5f | 409 | $ ustctl stop-trace 1234 auto |
1af1f1db PMF |
410 | |
411 | # Destroy the trace | |
eb78cc5f | 412 | $ ustctl destroy-trace 1234 auto |
5cd1099a PMF |
413 | @end verbatim |
414 | @end example | |
415 | ||
c3947c25 PMF |
416 | For more information about the manual mode, see the ustctl(1) man page. |
417 | ||
5cd1099a PMF |
418 | @node Using early tracing |
419 | @section Using early tracing | |
420 | ||
421 | Early tracing consists in starting the tracing as early as possible in the | |
422 | program, so no events are lost between program start and the point where the | |
423 | command to start the tracing is given. When using early tracing, it is | |
424 | guaranteed that by the time the traced program enters its @code{main()} | |
425 | function, the tracing will be started. | |
426 | ||
427 | When using @command{usttrace}, the early tracing is always active. | |
428 | ||
429 | When using the manual mode (@command{ustctl}), early tracing is enabled using | |
430 | environment variables. Setting @env{UST_TRACE} to @code{1}, enables early | |
431 | tracing, while setting @env{UST_AUTOPROBE} to @code{1} enables all markers | |
432 | automatically. | |
433 | ||
434 | ||
435 | @node Crash recovery | |
436 | @section Crash recovery | |
437 | ||
438 | When a process being traced crashes, the daemon is able to recover all the | |
439 | events in its buffers that were successfully commited. This is possible because | |
440 | the buffers are in a shared memory segment which remains available to the | |
441 | daemon even after the termination of the traced process. | |
442 | ||
443 | @node Tracing across @code{fork()} and @code{clone()} | |
444 | @section Tracing across @code{fork()} and @code{clone()} | |
445 | ||
2669948d PMF |
446 | Tracing across @code{clone()} when the @code{CLONE_VM} flag is specified is |
447 | supported without any particular action. | |
5cd1099a PMF |
448 | |
449 | When @code{clone()} is called without @code{CLONE_VM} or @code{fork()} is | |
450 | called, a new address space is created and the tracer must be notified to | |
2669948d PMF |
451 | create new buffers for it. |
452 | ||
453 | This can be done automatically, by @env{LD_PRELOAD}'ing @file{libinterfork.so}. | |
454 | This library intercepts calls to @code{fork()} and informs the tracer it is | |
455 | being called. When using @command{usttrace}, this is accomplied by specifying | |
456 | the @option{-f} command line argument. | |
457 | ||
458 | Alternatively, the program can call @code{ust_before_fork()} before calling | |
459 | @code{fork()} or @code{clone()} with @code{CLONE_VM}. After the call, | |
460 | @code{ust_after_fork_parent()} must be called in the parent process and | |
461 | @code{ust_after_fork_child()} must be called in the child process. | |
5cd1099a | 462 | |
5cd1099a PMF |
463 | |
464 | @node Tracing programs and libraries that were not linked to libust | |
465 | @section Tracing programs and libraries that were not linked to libust | |
466 | ||
2669948d PMF |
467 | Some programs need to be traced even though they were not linked to libust |
468 | either because they were not instrumented or because it was not practical. | |
469 | ||
470 | An executable that is not instrumented can still yield interesting traces when | |
471 | at least one of its dynamic libraries is instrumented. It is also possible to | |
472 | trace certain function calls by intercepting them with a specially crafted | |
473 | library that is linked with @env{LD_PRELOAD} at program start. | |
474 | ||
475 | In any case, a program that was not linked to libust at compile time must be | |
476 | linked to it at run time with @env{LD_PRELOAD}. This can be accomplished with | |
477 | @command{usttrace}'s @option{-l} option. It can also be done by setting the | |
478 | @env{LD_PRELOAD} environment variable on the command line. For example: | |
479 | ||
480 | @example | |
481 | @verbatim | |
482 | # Run ls with usttrace, LD_PRELOAD'ing libust | |
483 | # (assuming one of the libraries used by ls is instrumented). | |
484 | $ usttrace -l ls | |
485 | ||
486 | # Run ls, manually adding the LD_PRELOAD. | |
487 | $ LD_PRELOAD=/usr/local/lib/libust.so.0 ls | |
488 | @end verbatim | |
489 | @end example | |
490 | ||
5cd1099a PMF |
491 | |
492 | @node Performance | |
493 | @chapter Performance | |
494 | ||
495 | Todo. | |
496 | ||
497 | @node Viewing traces | |
498 | @chapter Viewing traces | |
499 | ||
2669948d PMF |
500 | Traces may be viewed with LTTV. An example of command for launching LTTV is |
501 | given in the quickstart. | |
502 | ||
503 | @menu | |
504 | * Viewing multiple traces:: | |
505 | * Combined kernel-userspace tracing:: | |
506 | @end menu | |
507 | ||
508 | @node Viewing multiple traces | |
509 | @section Viewing multiple traces | |
510 | ||
511 | When tracing multi-process applications or several applications simultaneously, | |
512 | more than one trace will be obtained. LTTV can open and display all these | |
513 | traces simultaneously. | |
514 | ||
515 | @node Combined kernel-userspace tracing | |
516 | @section Combined kernel-userspace tracing | |
517 | ||
518 | In addition to multiple userspace traces, LTTV can open a kernel trace recorded | |
519 | with the LTTng kernel tracer. This provides events that enable the rendering of | |
520 | the Control Flow View and the Resource View. | |
521 | ||
522 | When doing so, it is necessary to use the same time source for the kernel | |
523 | tracer as well as the userspace tracer. Currently, the recommended method is to | |
524 | use the timestamp counter for both. The TSC can however only be used on architectures | |
525 | where it is synchronized across cores. | |
526 | ||
bb0496b2 PMF |
527 | @node Resource Usage |
528 | @chapter Resource Usage | |
529 | ||
530 | The purpose of this section is to give an overview of the resource usage of libust. For | |
531 | a developer, knowing this can be important: because libust is linked with applications, it | |
532 | needs to share some resources with it. Some applications may make some assumptions that are in | |
533 | conflict with libust's usage of resources. | |
534 | ||
535 | In practice however, libust is designed to be transparent and is compatible | |
536 | with the vast majority of applications. This means no changes are required in | |
537 | the application (or library) being linked to libust. | |
538 | ||
4bea9b39 PMF |
539 | Libust is initialized by a constructor, which by definition runs before the |
540 | @code{main()} function of the application starts. This constructor creates a | |
541 | thread called the @emph{listener thread}. The listener thread initializes a | |
9dc7b7ff | 542 | named socket and waits for connections for ust-consumerd or ustctl. |
bb0496b2 PMF |
543 | |
544 | Libust-specific code may: | |
545 | @itemize @bullet | |
546 | @item use @code{malloc()} and @code{free()} | |
547 | @item map shared memory segment in the process adress space | |
548 | @item intercept some library calls, specifically @code{fork()} and @code{clone()} | |
549 | @item do interprocess communication with the daemon or ustctl | |
550 | @item create and open named sockets | |
551 | ||
552 | @end itemize | |
553 | ||
554 | It will not: | |
555 | @itemize @bullet | |
556 | @item handle any signal (all signals are blocked in the listener thread) | |
557 | @item change any process-wide setting that could confuse the application | |
558 | @end itemize | |
559 | ||
4bea9b39 PMF |
560 | @node List of environment variables detected by libust |
561 | @appendix List of environment variables detected by libust | |
562 | ||
563 | The behavior of tracing can be influenced by setting special environment | |
564 | variables in the environment of the traced application. This section | |
565 | describes these variables. | |
566 | ||
567 | @itemize @bullet | |
568 | ||
569 | @item | |
570 | @env{UST_TRACE} | |
571 | ||
572 | If set to 1, start tracing as soon as the program starts. Tracing is | |
573 | guaranteed to be started by the time the @code{main()} function starts. | |
574 | ||
575 | @item | |
576 | @env{UST_AUTOPROBE} | |
577 | ||
578 | If set to @code{1}, enable all markers by the time the @code{main()} function starts. | |
579 | ||
580 | @item | |
581 | @env{UST_AUTOCOLLECT} | |
582 | ||
583 | If set to @code{0}, disable notification of daemon on trace start. Useful for | |
584 | performance tests. | |
585 | ||
586 | @item | |
587 | @env{UST_OVERWRITE} | |
588 | ||
589 | If set to @code{1}, enable overwriting of buffers on overrun. | |
590 | ||
591 | @item | |
592 | @env{UST_SUBBUF_NUM} | |
593 | ||
594 | If set, defines the default number of subbuffers per buffer. | |
595 | ||
596 | @item | |
597 | @env{UST_SUBBUF_SIZE} | |
598 | ||
599 | If set, defines the default size of subbuffers, in bytes. | |
600 | ||
601 | @end itemize | |
602 | ||
603 | @node GDB integration | |
604 | @appendix GDB integration | |
605 | ||
606 | GDB, the GNU Debugger, can use UST markers as GDB tracepoints (note GDB has its | |
607 | own concept of tracepoint). This feature is called GDB Static Tracepoints. When | |
608 | a GDB tracepoint is hit, GDB collects the marker arguments, as well as the | |
fe566790 | 609 | state of the registers. Support for GDB is currently work in progress. |
bb0496b2 | 610 | |
5cd1099a | 611 | @bye |