[lttv.git] / ltt / branches / poly / doc / developer / format.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/2002/REC-xhtml1-20020801/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

<head>
  <title>The LTTng trace format</title>
</head>

<body>

<h1>The LTTng trace format</h1>

<p>
<em>Last update: 2008/06/02</em>
</p>

<p>
This document describes the LTTng trace format. It should be useful mainly to
developers who code the LTTng tracer or the traceread LTTV library, as this
library offers all the necessary abstractions on top of the raw trace data.
</p>

<p>
A trace is contained in a directory tree. To send a trace remotely, the
directory tree may be tar-gzipped. The trace <tt>foo</tt>, placed in the home
directory of user john, /home/john, would have the following contents:
</p>

<pre><tt>
$ cd /home/john
$ tree foo
foo/
|-- control
|   |-- facilities_0
|   |-- facilities_1
|   |-- facilities_...
|   |-- interrupts_0
|   |-- interrupts_1
|   |-- interrupts_...
|   |-- modules_0
|   |-- modules_1
|   |-- modules_...
|   `-- processes_0
|   `-- processes_1
|   `-- processes_...
|-- cpu_0
|-- cpu_1
`-- cpu_...

</tt></pre>

<p>
The root directory contains a tracefile for each cpu, numbered from 0, 
in .trace format. A uniprocessor thus only contains the file cpu_0. 
A multi-processor with some unused (possibly hotplug) CPU slots may have some
unused CPU numbers. For instance an 8 way SMP board with 6 CPUs randomly 
installed may produce tracefiles named 0, 1, 2, 4, 6, 7.
</p>

<p>
The files in the control directory also follow the .trace format and are
also per cpu.  The "facilities" files only contain "core" marker_id,
marker_format and time_heartbeat events. The first two are used to describe the
events that are in the trace. The other control files contain the initial
system state and various subsequent important events, for example process
creations and exit. The interest of placing such subsequent events in control
trace files instead of (or in addition to) in the per cpu trace files is that
they may be accessed more quickly/conveniently and that they may be kept even
when the per cpu files are overwritten in "flight recorder mode".
</p>

<h2>Trace format</h2>

<p>
Each tracefile is divided into equal size blocks with a header at the beginning
of the block. Events are packed sequentially in the block starting right after
the block header.
</p>

<p>
Each block consists of :
</p>

<pre><tt>
block start/end header
trace header
event 1 header
event 1 variable length data
event 2 header
event 2 variable length data
....
padding
</tt></pre>

<h3>The block start/end header</h3>

<pre><tt>
begin
	* the beginning of buffer information
	uint64 cycle_count
		* TSC at the beginning of the buffer
	uint64 freq
		* frequency of the CPUs at the beginning of the buffer.
end
	* the end of buffer information
	uint64 cycle_count
		* TSC at the end of the buffer
	uint64 freq
		* frequency of the CPUs at the end of the buffer.
uint32 lost_size
	* number of bytes of padding at the end of the buffer.
uint32 buf_size
	* size of the sub-buffer.
</tt></pre>


<h3>The trace header</h3>

<pre><tt>
uint32 magic_number
	* 0x00D6B7ED, used to check the trace byte order vs host byte order.
uint32 arch_type
	* Architecture type of the traced machine.
uint32 arch_variant
	* Architecture variant of the traced machine. May be unused on some arch.
uint32 float_word_order
	* Byte order of floats and doubles, sometimes different from integer byte
	  order. Useful only for user space traces.
uint8 arch_size
	* Size (in bytes) of the void * on the traced machine.
uint8 major_version
	* major version of the trace.
uint8 minor_version
	* minor version of the trace.
uint8 flight_recorder
	* Is flight recorder mode activated ? If yes, data might be missing
	  (overwritten) in the trace.
uint8	has_heartbeat
	* Does this trace have heartbeat timer event activated ?
		Yes (1) -> Event header has 32 bits TSC
		No (0) -> Event header has 64 bits TSC
uint8 alignment
	* Are event headers in this trace aligned ?
		Yes -> the value indicates the alignment
		No (0) -> data is packed.
uint8 tsc_lsb_truncate
uint8 tscbits
uint8 compact_data_shift
uint32 freq_scale
		event time is always calculated from :
			trace_start_time + ((event_tsc - trace_start_tsc) * (freq / freq_scale))
uint64 start_freq
	* CPUs clock frequency at the beginnig of the trace.
uint64 start_tsc
	* TSC at the beginning of the trace.
uint64 start_monotonic
	* monotonically increasing time at the beginning of the trace.
		(currently not supported)
start_time
	* Real time at the beginning of the trace (as given by date, adjusted by NTP)
		This is the only time reference with the real world : the rest of the trace
		has monotonically increasing time from this point (with TSC difference and
		clock frequency).
	uint32 seconds
	uint32 nanoseconds
</tt></pre>


<h3>Event header</h3>

<p>
Event headers differ according to the following conditions : does the
traced system have a heartbeat timer? Is tracing alignment activated?
</p>

<p>
Event header :
</p>
<pre><tt>
{ uint32 timestamp
	or
	uint64 timestamp }
	* if has_heartbeat : 32 LSB of the cycle counter at the event record time.
	* else : 64 bits complete cycle counter.
uint8 facility_id
	* Numerical ID of the facility corresponding to the event. See the facility
	  tracefile to know which facility ID matches which facility name and
		description.
uint8 event_id
	* Numerical ID of the event inside the facility.
uint16 event_size
	* Size of the variable length data that follows this header.
</tt></pre>

<p>
Event header alignment
</p>

<p>
If trace alignment is activated (<tt>alignment</tt>), the event header is
aligned. In addition, padding is automatically added after the event header so
the variable length data is automatically aligned on the architecture size.
</p>

<!--
<h2>System description</h2>

<p>
The system type description, in system.xml, looks like:
</p>

<pre><tt>
&lt;system 
 node_name="vaucluse"
 domainname="polymtl.ca" 
 cpu=4
 arch_size="ILP32" 
 endian="little" 
 kernel_name="Linux" 
 kernel_release="2.4.18-686-smp" 
 kernel_version="#1 SMP Sun Apr 14 12:07:19 EST 2002"
 machine="i686" 
 processor="unknown" 
 hardware_platform="unknown"
 operating_system="Linux" 
 ltt_major_version="2"
 ltt_minor_version="0"
 ltt_block_size="100000"
&gt;
Some comments about the system
&lt;/system&gt;
</tt></pre>

<p>
The system attributes kernel_name, node_name, kernel_release,
 kernel_version, machine, processor, hardware_platform and operating_system
come from the uname(1) program. The domainname attribute is obtained from
the "hostname &#045;&#045;domain" command. The arch_size attribute is one of
LP32, ILP32, LP64 or ILP64 and specifies the length in bits of integers (I),
long (L) and pointers (P). The endian attribute is "little" or "big".
While the arch_size and endian attributes could be deduced from the platform
type, having these explicit allows analysing traces from yet unknown
platforms. The cpu attribute specifies the maximum number of processors in
the system; only tracefiles 0 to this maximum - 1 may exist in the cpu
directory.
</p>

<p>
Within the system element, the text enclosed may describe further the
system traced.
</p>


<h2>Bookmarks</h2>

<p>
Bookmarks are user supplied information added to a trace. They contain user
annotations attached to a time interval.
</p>


<pre><tt>
&lt;bookmarks&gt;
  &lt;location name=name cpu=n start_time=t end_time=t&gt;Some text&lt;/location&gt;
  ...
&lt;/bookmarks&gt;
</tt></pre>

<p>
The interval is defined using either "time=" or "start_time=" and 
"end_time=", or "cycle=" or "start_cycle=" and "end_cycle=". 
The time is in seconds with decimals up to nanoseconds and cycle counts 
are unsigned integers with a 64 bits range. The cpu attribute is optional.
</p>

-->
</body>
</html>
Commit	Line	Data
1f39128c	1	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/2002/REC-xhtml1-20020801/DTD/xhtml1-strict.dtd">
	2	<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	3
584db146	4	<head>
a25fb9c4	5	<title>The LTTng trace format</title>
584db146	6	</head>
1f39128c	7
1f39128c	8	<body>
584db146	9
a25fb9c4	10	<h1>The LTTng trace format</h1>
a25fb9c4	11
1f39128c	12	<p>
	13	<em>Last update: 2008/06/02</em>
	14	</p>
147c6f7f	15
1f39128c	16	<p>
147c6f7f	17	This document describes the LTTng trace format. It should be useful mainly to
a25fb9c4	18	developers who code the LTTng tracer or the traceread LTTV library, as this
a25fb9c4	19	library offers all the necessary abstractions on top of the raw trace data.
1f39128c	20	</p>
584db146	21
1f39128c	22	<p>
147c6f7f	23	A trace is contained in a directory tree. To send a trace remotely, the
	24	directory tree may be tar-gzipped. The trace <tt>foo</tt>, placed in the home
	25	directory of user john, /home/john, would have the following contents:
1f39128c	26	</p>
584db146	27
1f39128c	28	<pre><tt>
584db146	29	$ cd /home/john
	30	$ tree foo
	31	foo/
584db146	32	\|-- control
a25fb9c4	33	\| \|-- facilities_0
	34	\| \|-- facilities_1
	35	\| \|-- facilities_...
	36	\| \|-- interrupts_0
	37	\| \|-- interrupts_1
	38	\| \|-- interrupts_...
	39	\| \|-- modules_0
	40	\| \|-- modules_1
	41	\| \|-- modules_...
	42	\| `-- processes_0
	43	\| `-- processes_1
	44	\| `-- processes_...
	45	\|-- cpu_0
	46	\|-- cpu_1
	47	`-- cpu_...
	48
1f39128c	49	</tt></pre>
584db146	50
1f39128c	51	<p>
a25fb9c4	52	The root directory contains a tracefile for each cpu, numbered from 0,
a25fb9c4	53	in .trace format. A uniprocessor thus only contains the file cpu_0.
584db146	54	A multi-processor with some unused (possibly hotplug) CPU slots may have some
147c6f7f	55	unused CPU numbers. For instance an 8 way SMP board with 6 CPUs randomly
584db146	56	installed may produce tracefiles named 0, 1, 2, 4, 6, 7.
1f39128c	57	</p>
584db146	58
1f39128c	59	<p>
147c6f7f	60	The files in the control directory also follow the .trace format and are
	61	also per cpu. The "facilities" files only contain "core" marker_id,
	62	marker_format and time_heartbeat events. The first two are used to describe the
	63	events that are in the trace. The other control files contain the initial
	64	system state and various subsequent important events, for example process
	65	creations and exit. The interest of placing such subsequent events in control
	66	trace files instead of (or in addition to) in the per cpu trace files is that
	67	they may be accessed more quickly/conveniently and that they may be kept even
	68	when the per cpu files are overwritten in "flight recorder mode".
1f39128c	69	</p>
584db146	70
1f39128c	71	<h2>Trace format</h2>
584db146	72
1f39128c	73	<p>
a25fb9c4	74	Each tracefile is divided into equal size blocks with a header at the beginning
	75	of the block. Events are packed sequentially in the block starting right after
	76	the block header.
1f39128c	77	</p>
	78
	79	<p>
a25fb9c4	80	Each block consists of :
1f39128c	81	</p>
	82
	83	<pre><tt>
a25fb9c4	84	block start/end header
	85	trace header
	86	event 1 header
	87	event 1 variable length data
	88	event 2 header
	89	event 2 variable length data
	90	....
	91	padding
1f39128c	92	</tt></pre>
a25fb9c4	93
1f39128c	94	<h3>The block start/end header</h3>
a25fb9c4	95
1f39128c	96	<pre><tt>
a25fb9c4	97	begin
a25fb9c4	98	* the beginning of buffer information
a25fb9c4	99	uint64 cycle_count
	100	* TSC at the beginning of the buffer
	101	uint64 freq
	102	* frequency of the CPUs at the beginning of the buffer.
	103	end
	104	* the end of buffer information
a25fb9c4	105	uint64 cycle_count
147c6f7f	106	* TSC at the end of the buffer
a25fb9c4	107	uint64 freq
d88e4d7c	108	* frequency of the CPUs at the end of the buffer.
a25fb9c4	109	uint32 lost_size
	110	* number of bytes of padding at the end of the buffer.
	111	uint32 buf_size
	112	* size of the sub-buffer.
1f39128c	113	</tt></pre>
a25fb9c4	114
	115
	116
1f39128c	117	<h3>The trace header</h3>
a25fb9c4	118
1f39128c	119	<pre><tt>
a25fb9c4	120	uint32 magic_number
	121	* 0x00D6B7ED, used to check the trace byte order vs host byte order.
	122	uint32 arch_type
	123	* Architecture type of the traced machine.
	124	uint32 arch_variant
	125	* Architecture variant of the traced machine. May be unused on some arch.
	126	uint32 float_word_order
	127	* Byte order of floats and doubles, sometimes different from integer byte
	128	order. Useful only for user space traces.
	129	uint8 arch_size
	130	* Size (in bytes) of the void * on the traced machine.
	131	uint8 major_version
	132	* major version of the trace.
	133	uint8 minor_version
	134	* minor version of the trace.
	135	uint8 flight_recorder
	136	* Is flight recorder mode activated ? If yes, data might be missing
	137	(overwritten) in the trace.
	138	uint8 has_heartbeat
	139	* Does this trace have heartbeat timer event activated ?
	140	Yes (1) -> Event header has 32 bits TSC
	141	No (0) -> Event header has 64 bits TSC
147c6f7f	142	uint8 alignment
	143	* Are event headers in this trace aligned ?
	144	Yes -> the value indicates the alignment
a25fb9c4	145	No (0) -> data is packed.
147c6f7f	146	uint8 tsc_lsb_truncate
	147	uint8 tscbits
	148	uint8 compact_data_shift
99f2111d	149	uint32 freq_scale
cb310b57	150	event time is always calculated from :
cb310b57	151	trace_start_time + ((event_tsc - trace_start_tsc) * (freq / freq_scale))
a25fb9c4	152	uint64 start_freq
	153	* CPUs clock frequency at the beginnig of the trace.
	154	uint64 start_tsc
	155	* TSC at the beginning of the trace.
	156	uint64 start_monotonic
	157	* monotonically increasing time at the beginning of the trace.
	158	(currently not supported)
	159	start_time
	160	* Real time at the beginning of the trace (as given by date, adjusted by NTP)
	161	This is the only time reference with the real world : the rest of the trace
	162	has monotonically increasing time from this point (with TSC difference and
	163	clock frequency).
	164	uint32 seconds
	165	uint32 nanoseconds
1f39128c	166	</tt></pre>
a25fb9c4	167
584db146	168
1f39128c	169	<h3>Event header</h3>
584db146	170
1f39128c	171	<p>
147c6f7f	172	Event headers differ according to the following conditions : does the
147c6f7f	173	traced system have a heartbeat timer? Is tracing alignment activated?
1f39128c	174	</p>
a25fb9c4	175
1f39128c	176	<p>
a25fb9c4	177	Event header :
1f39128c	178	</p>
1f39128c	179	<pre><tt>
a25fb9c4	180	{ uint32 timestamp
	181	or
	182	uint64 timestamp }
	183	* if has_heartbeat : 32 LSB of the cycle counter at the event record time.
	184	* else : 64 bits complete cycle counter.
a25fb9c4	185	uint8 facility_id
	186	* Numerical ID of the facility corresponding to the event. See the facility
	187	tracefile to know which facility ID matches which facility name and
	188	description.
	189	uint8 event_id
	190	* Numerical ID of the event inside the facility.
	191	uint16 event_size
	192	* Size of the variable length data that follows this header.
1f39128c	193	</tt></pre>
a25fb9c4	194
1f39128c	195	<p>
a25fb9c4	196	Event header alignment
1f39128c	197	</p>
a25fb9c4	198
1f39128c	199	<p>
147c6f7f	200	If trace alignment is activated (<tt>alignment</tt>), the event header is
	201	aligned. In addition, padding is automatically added after the event header so
	202	the variable length data is automatically aligned on the architecture size.
1f39128c	203	</p>
a25fb9c4	204
147c6f7f	205	<!--
1f39128c	206	<h2>System description</h2>
584db146	207
1f39128c	208	<p>
584db146	209	The system type description, in system.xml, looks like:
1f39128c	210	</p>
584db146	211
1f39128c	212	<pre><tt>
584db146	213	<system
	214	node_name="vaucluse"
	215	domainname="polymtl.ca"
	216	cpu=4
	217	arch_size="ILP32"
	218	endian="little"
	219	kernel_name="Linux"
	220	kernel_release="2.4.18-686-smp"
	221	kernel_version="#1 SMP Sun Apr 14 12:07:19 EST 2002"
	222	machine="i686"
	223	processor="unknown"
	224	hardware_platform="unknown"
	225	operating_system="Linux"
	226	ltt_major_version="2"
	227	ltt_minor_version="0"
	228	ltt_block_size="100000"
	229	>
	230	Some comments about the system
	231	</system>
1f39128c	232	</tt></pre>
584db146	233
1f39128c	234	<p>
584db146	235	The system attributes kernel_name, node_name, kernel_release,
	236	kernel_version, machine, processor, hardware_platform and operating_system
	237	come from the uname(1) program. The domainname attribute is obtained from
147c6f7f	238	the "hostname --domain" command. The arch_size attribute is one of
584db146	239	LP32, ILP32, LP64 or ILP64 and specifies the length in bits of integers (I),
	240	long (L) and pointers (P). The endian attribute is "little" or "big".
	241	While the arch_size and endian attributes could be deduced from the platform
	242	type, having these explicit allows analysing traces from yet unknown
	243	platforms. The cpu attribute specifies the maximum number of processors in
	244	the system; only tracefiles 0 to this maximum - 1 may exist in the cpu
	245	directory.
1f39128c	246	</p>
584db146	247
1f39128c	248	<p>
584db146	249	Within the system element, the text enclosed may describe further the
584db146	250	system traced.
1f39128c	251	</p>
584db146	252
584db146	253
1f39128c	254	<h2>Bookmarks</h2>
	255
	256	<p>
584db146	257	Bookmarks are user supplied information added to a trace. They contain user
584db146	258	annotations attached to a time interval.
1f39128c	259	</p>
1f39128c	260
584db146	261
1f39128c	262	<pre><tt>
584db146	263	<bookmarks>
	264	<location name=name cpu=n start_time=t end_time=t>Some text</location>
	265	...
	266	</bookmarks>
1f39128c	267	</tt></pre>
584db146	268
1f39128c	269	<p>
584db146	270	The interval is defined using either "time=" or "start_time=" and
	271	"end_time=", or "cycle=" or "start_cycle=" and "end_cycle=".
	272	The time is in seconds with decimals up to nanoseconds and cycle counts
	273	are unsigned integers with a 64 bits range. The cpu attribute is optional.
1f39128c	274	</p>
1f39128c	275
147c6f7f	276	-->
1f39128c	277	</body>
1f39128c	278	</html>