Tracing is routinely used to help understanding the behavior and performance of various aspects of the Linux kernel and associated drivers. Many of the 80K+ printk statements in the Linux kernel serve this purpose, although printk is relatively low performance and unreliable. The small default printk buffer size coupled with the low performance brings lost messages as soon as the volume becomes significant.
For this reason, a number of drivers include their own tracing macros and infrastructure. A quick search looking for TRACE and related keywords in the Linux kernel source reveals some form of tracing in at least the following files:
A number of tracing tools have been developed for the Linux kernel. The best known, particularly in the embedded systems area, is the Linux Trace Toolkit, LTT at http://www.opersys.com/LTT. It comes with a nice graphical user interface and is currently under active development to add dynamically defined event types and graphical trace analysis modules.
The Linux Kernel State Tracer at http://lkst.sf.netwas developed by Hitachi and offers basic, low overhead, tracing functionality. There is no grahical user interface available.
MAGNET was recently released. It was initially developed to trace the network stack and drivers. Its performance has not been optimized for SMP systems. It is available from http://public.lanl.gov/radiant/software/magnet.html .
The IKD patch from Andrea Arcangeli ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/ includes ktrace which adds the -pg gcc compilation option to specified source files. This adds a call to function mcount upon entry in any function compiled with that option. A function mcount is provided which records in a trace the address of the function entered. Using the system map, this is later translated into a trace of names of functions entered.
Tracing may be placed in the larger context of Reliability, Availability and Serviceability (RAS). The Linux RAS project is probably the most active and well organized, http://systemras.sourceforge.net/ http://www-124.ibm.com/linux/projects/linuxras/ . It links to several underlying projects, including the Linux Trace Toolkit LTT.
Several other projects within Linux RAS directly relate to tracing.
The Enterprise Event Logging project, EVLOG project at http://evlog.sourceforge.net/, produces traces and thus shares a number of underlying implementation needs (events recording, kernel to user mode transfer, trace analysis and viewing tools, event types format). The intended purpose and thus implementation constraints differ significantly, however. EVLOG records important system events for two purposes, to trigger service and security alarms (e.g. weak signals in a magnetic disk, unauthorized access attempt) and to provide permament records. The volume is typically low and full context is required for each event. While logging (EVLOG) is therefore implemented separately from tracing (LTT), some underlying technology may be reused as appropriate (e.g. kernel hooks, kernel to user mode data relay...).
A common symptom of a serious kernel problem is a crash. Traces may be extremely useful to understand the problem except that, because of the crash, the important last events in the current trace buffer cannot be stored on disk. The Linux Kernel Crash Dump facility (LKCD) at http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/ is used to recover such information, when warm rebooting from a crash while this information is still available in memory.
LKCD needs to be told how to find the tracing buffers in the memory (address in a map or signature to look for) and in which file to save their content.
Kernel hooks, at http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/ are a mechanism to insert hooks at desired locations in the kernel. Handlers may later be registered to be called at these hooks locations. When no handler is registered, the cost associated with a hook is almost negligeable, a few NOPs. Skipping NOPs is even faster than testing a global boolean variable. Kernel hooks would be ideally suited for the dynamic activation of trace points. Furthermore, kernel hooks allow registering multiple handlers. A same location could have a tracing handler and a performance tool handler, reducing the number of points needed to be inserted in the kernel source code.
Interactive tools may be used to rapidly select groups of hooks to be activated based on facilities (networking, block devices...), level of details (core events, detailed events) or severity level (warning, info, debug).
As part of Kernel Hooks and Dynamic Probes, were defined handlers which produce tracing information. The tracing data models for Dynamic Probes and LTT are fairly similar and may eventually be consolidated.
The Dynamic Probes, http://www-124.ibm.com/linux/projects/kprobes/ , allow inserting kernel hooks dynamically in a running kernel, just like breakpoints in debuggers. The instruction at the desired location is saved and replaced by an interrupt instruction. When the interrupt instruction is executed, the handlers are called, the original instruction restored and executed in single step mode, and the interrupt instruction is reinserted.