lttv/doc/developer/tracing_tools_overview.html

   1 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
   2 <html>
   3 <head>
   4   <title>Tracing Tools</title>
   5 </head>
   6   <body>
   7
   8 <h1>Tracing Tools</h1>
   9
  10 <p>Tracing is routinely used to help understanding the behavior and performance
  11 of various aspects of the Linux kernel and associated drivers.
  12 Many of the 80K+ printk statements in the Linux kernel
  13 serve this purpose, although printk is relatively low
  14 performance and unreliable. The small default printk buffer size coupled with
  15 the low performance brings lost messages as soon as the volume becomes
  16 significant.
  17
  18 <p>For this reason, a number of drivers include their own tracing macros
  19 and infrastructure. A quick search looking for TRACE and related keywords
  20 in the Linux kernel source reveals some form of tracing in at least
  21 the following files:
  22
  23 <UL>
  24 <LI>./fs/hpfs/hpfs_fn.h
  25 <LI>./fs/smbfs/smb_debug.h
  26 <LI>./fs/autofs/autofs_i.h
  27 <LI>./fs/jffs2/nodelist.h
  28 <LI>./include/linux/wait.h
  29 <LI>./include/linux/parport_pc.h
  30 <LI>./include/linux/amigaffs.h
  31 <LI>./include/linux/parport_pc.h
  32 <LI>./include/linux/ncp_fs.h
  33 <LI>drivers/net/wireless airport and orinoco
  34 <LI>drivers/char/ftape
  35 <LI>drivers/char/dtlk.c
  36 <LI>drivers/char/mwave
  37 <LI>drivers/char/n_r3964.c
  38 <LI>drivers/scsi/qlogicfc.c
  39 <LI>drivers/usb/pwc-if.c
  40 <LI>drivers/usb/hpusbscsi.c
  41 <LI>drivers/acpi/include/acmacros.h
  42 <LI>arch/sparc/kernel/signal.c
  43 <LI>arch/mips/math-emu/cp1emu.c
  44 <LI>drivers/net/wavelan.c
  45 <LI>drivers/net/hp100.c
  46 <LI>drivers/net/wan/lmc/lmc_debug.c
  47 <LI>drivers/net/skfp/h/targetos.h
  48 <LI>drivers/char/ip2main.c
  49 <LI>drivers/scsi/gdth.c
  50 <LI>drivers/scsi/megaraid.c
  51 <LI>drivers/scsi/qlogicisp.c
  52 <LI>drivers/scsi/ips.c
  53 <LI>drivers/scsi/qla1280.c
  54 <LI>drivers/scsi/cpqfcTSstructs.h
  55 <LI>drivers/cdrom/sjcd.c
  56 <LI>drivers/isdn/eicon/sys.h
  57 <LI>drivers/sbus/char/bbc_envctrl.c
  58 <LI>drivers/ide/ide-tape.c
  59 <LI>drivers/video/radeonfb.c
  60 <LI>fs/intermezzo/sysctl.c
  61 <LI>fs/ext3/balloc.c
  62 <LI>net/ipv6/ip6_fib.c
  63 <LI>net/irda/irnet/irnet.h
  64 <UL>
  65
  66 <p>A number of tracing tools have been developed for the Linux kernel.
  67 The best known, particularly in the embedded systems area, is the Linux Trace
  68 Toolkit, <A HREF="http://www.opersys.com/LTT">LTT at
  69 http://www.opersys.com/LTT</A>. It
  70 comes with a nice graphical user interface and is currently under active
  71 development to add dynamically defined event types and graphical trace
  72 analysis modules.
  73
  74 <P>
  75 The <A HREF="http://lkst.sf.net">Linux Kernel State Tracer at
  76 http://lkst.sf.net</A>was developed by Hitachi and offers basic,
  77 low overhead, tracing functionality. There is no grahical user interface
  78 available.
  79
  80 <P>
  81 MAGNET was recently released. It was initially developed to trace the network
  82 stack and drivers. Its performance has not been optimized for SMP systems.
  83 It is available from
  84 <A HREF="http://public.lanl.gov/radiant/software/magnet.html">
  85 http://public.lanl.gov/radiant/software/magnet.html
  86 </A>.
  87
  88 <P>
  89 The IKD patch from Andrea Arcangeli
  90 <A HREF="ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/">
  91 ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/
  92 </A>
  93 includes ktrace which adds the -pg gcc compilation option
  94 to specified source files. This adds a call to function <i>mcount</i>
  95 upon entry in any function compiled with that option. A function <i>mcount</i>
  96 is provided which records in a trace the address of the function entered.
  97 Using the system map, this is later translated into a trace of names of
  98 functions entered.
  99
 100 <H2>Reliability, Availability and Serviceability</H2>
 101
 102 <P>
 103 Tracing may be placed in the larger context of Reliability, Availability and
 104 Serviceability (RAS). The Linux RAS project is probably the most active and
 105 well organized,
 106 <A HREF="http://systemras.sourceforge.net/">
 107 http://systemras.sourceforge.net/
 108 </A>
 109 <A HREF="http://www-124.ibm.com/linux/projects/linuxras/">
 110 http://www-124.ibm.com/linux/projects/linuxras/
 111 </A>.
 112 It links to several underlying projects, including the Linux Trace Toolkit
 113 <A HREF="http://www.opersys.com/LTT">LTT</A>.
 114
 115 <P>
 116 Several other projects within Linux RAS directly relate to tracing.
 117
 118 <H3>Enterprise Event Logging</H3>
 119
 120 <p>The Enterprise Event Logging project,
 121 <A HREF="http://evlog.sourceforge.net/">EVLOG project
 122 at http://evlog.sourceforge.net/</A>, produces traces and thus shares a number
 123 of underlying implementation needs
 124 (events recording, kernel to user mode transfer,
 125 trace analysis and viewing tools, event types format). The intended purpose
 126 and thus implementation constraints differ significantly, however.
 127 EVLOG records important system events for two purposes,
 128 to trigger service and security alarms (e.g. weak signals in a magnetic disk,
 129 unauthorized access attempt) and to provide permament records. The volume
 130 is typically low and full context is required for each event. While logging
 131 (EVLOG) is therefore implemented separately from tracing (LTT), some
 132 underlying technology may be reused as appropriate (e.g. kernel hooks,
 133 kernel to user mode data relay...).
 134
 135 <H3>Kernel Crash Dump</H3>
 136
 137 <P>A common symptom of a serious kernel problem is a crash. Traces may
 138 be extremely useful to understand the problem except that, because of the
 139 crash, the important last events in the current trace buffer cannot be
 140 stored on disk. The Linux Kernel Crash Dump facility (LKCD) at
 141 <A HREF="http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/">
 142 http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/
 143 </A> is used to recover such information, when <i>warm</i> rebooting from a
 144 crash while this information is still available in memory.
 145
 146 <P>LKCD needs to be told how to find the tracing buffers in the memory
 147 (address in a map or signature to look for) and in which file to save
 148 their content.
 149
 150 <H3>Kernel Hooks</H3>
 151
 152 <p>
 153 Kernel hooks, at
 154 <A HREF="http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/">
 155 http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/
 156 </A> are a mechanism to insert hooks at desired locations in the kernel.
 157 Handlers may later be registered to be called at these hooks locations.
 158 When no handler is registered, the cost associated with a hook is almost
 159 negligeable, a few NOPs. Skipping NOPs is even faster than testing a
 160 global boolean variable. Kernel hooks would be ideally suited for the
 161 dynamic activation of trace points. Furthermore, kernel hooks allow registering
 162 multiple handlers. A same location could have a tracing handler and a
 163 performance tool handler, reducing the number of points needed to be
 164 inserted in the kernel source code.
 165
 166 <p>Interactive tools may be used to rapidly select groups of hooks to be
 167 activated based on facilities (networking, block devices...), level
 168 of details (core events, detailed events) or severity level (warning, info,
 169 debug).
 170
 171 <p>As part of Kernel Hooks and Dynamic Probes, were defined handlers
 172 which produce tracing information. The tracing data models for Dynamic Probes
 173 and LTT are fairly similar and may eventually be consolidated.
 174
 175 <H3>Dynamic Probes</H3>
 176
 177 <p>The Dynamic Probes,
 178 <A HREF="http://www-124.ibm.com/linux/projects/kprobes/">
 179 http://www-124.ibm.com/linux/projects/kprobes/
 180 </A>,
 181 allow inserting kernel hooks dynamically in a running kernel, just like
 182 breakpoints in debuggers. The instruction
 183 at the desired location is saved and replaced by an interrupt instruction.
 184 When the interrupt instruction is executed, the handlers are called, the
 185 original instruction restored and executed in single step mode, and the
 186 interrupt instruction is reinserted.
 187
 188 </body>
 189 </html>