Add lttng URL proposal for the public API
[lttng-tools.git] / doc / proposals / 0001-session-daemon-architecture.txt
CommitLineData
fa8f9c82
DG
1RFC - LTTng session daemon architecture
2
3WARNING:
4 Parts of the proposal are obselete but we keep this version for historical
5 purposes.
508f6562
DG
6
7Author: David Goulet <david.goulet@polymtl.ca>
8
9Contributors:
10 * Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
11 * Yannick Brosseau <yannick.brosseau@polymtl.ca>
12 * Nils Carlson <nils.carlson@ericsson.com>
13 * Michel Dagenais <michel.dagenais@polymtl.ca>
14 * Stefan Hajnoczi <stefanha@gmail.com>
15
16Version:
17 - v0.1: 17/01/2011
18 * Initial proposal
19
fa8f9c82 20 - v0.2: 19/01/2011
508f6562
DG
21 After multiple reply from all the contributors above, here is the list
22 of what has changed:
23 * Change/Add Terminology elements from the initial model
24 * New figures for four new scenarios
25 * Add inprocess library section
26 * LTTng kernel tracer support proposition
27 * More details for the Model and Components
28 * Improve the basic model. Quite different from the last one
fa8f9c82 29
508f6562
DG
30 - v0.3: 28/01/2011
31 In response from Michel Dagenais and Nils Carlson comments:
32 * Add scaling to reasons of this re-engineering
33 * Purpose of the session ID
34 * Explain why ltt-sessiond creates the tracing buffers
35 * ust-consumerd interaction schema
36 * Clarify inprocess library behavior
37
38 - v0.4: 01/02/2011
39 After Mathieu Desnoyers and Michel Dagenais comments:
40 * Add section Introduction
fa8f9c82 41 * Define the global and per-user ltt-sessiond
508f6562
DG
42 * Add details for ltt-sessiond in the inprocess lib section
43 * Session ID are now UUID
44 * Add buffer snapshot schema for ust-consumerd
45 * ltt-sessiond validate inprocess lib version
46 * ltt-sessiond socket validation by the inprocess lib.
47 * Add lttng definition
48 * Add consumer to the Model section
fa8f9c82
DG
49 - v0.5: 23/07/2012
50 * Please refer to the thesis of David Goulet for the complete
51 and up to date specifications of the architecture and algorithms used.
52 (http://publications.polymtl.ca/842/1/2012_DavidGoulet.pdf)
508f6562
DG
53
54Terminology
55-----------------
56
57ltt-sessiond - Main daemon for trace session registry for UST and LTTng
32258573 58NOTE: Changed to lttng-sessiond in the git tree
508f6562
DG
59
60ust-consumerd - Daemon that consume UST buffers for a specific application
61
62ltt-consumerd - Daemon that consume LTTng buffers
63
64tracing session - A trace linked to a set of specific tracepoints and to a set
65 of tracing buffers
66
67tracing buffers - Buffers containing tracing data
68
69tracing data - Data created by tracing an application
70
71inprocess library - UST library linked with the application
72
73shared memory - system V shared memory
74
75application common named pipe - Global named pipe that triggers application
76 registration, on pipe event, to ltt-sessiond
77
78lttng - New command line tool for LTTng and UST tracing control
79
80Introduction
81-----------------
82
83This RFC propose a brand new UST and LTTng daemon model. This re-engineering
84was mostly driven by the need of:
85
86 * Better security in terms of access rights on tracing data
87 * Manage tracing session
88 * Scaling in terms of thread/processes needed to perform tracing
89 * LTTng and UST integration in terms of merging traces and session control
90 * Networking such as streaming and remote control over different traces
91
92The new model follows the basic principles of having a session registry
93(ltt-sessiond) and consumers for each tracing session (ust-consumerd and
94ltt-consumerd).
95
96With this proposal, LTTng and UST will share the same tracing session, be
97managed by the same tool and bring a complete integration between these two
98powerful tools.
99
100NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa.
101
102Model
103-----------------
104
105A global and/or per-user registry keeps track of all tracing sessions. Any user
106that wants to manage either a kernel trace using LTTng or an application trace
107with UST must interact with that registry for any possible actions.
108
109The model address multiple tracing use cases based on the fact that we
110introduce a tracing Unix group (tracing group). Only users in that group or
111root can use the global registry. Other users will create a local registry
112(per-user registry) that will be completely independent from the global one.
113
114Two cases:
115
116 1) Users in the tracing group, it's tracing session can consume all tracing
117 buffers from all applications and the kernel.
118
119 2) Users NOT in the tracing group, it's tracing session can only consume
120 data from its own applications' buffers hence tracing his applications.
121
122A session stored by the registry consist of:
123
124 * Session name (given by the user or automatically assigned)
125 * List of traces (LTTng or UST)
126 * Tracepoints/markers associated to a trace of that session
127 * UUID
128 * Associated user (UID)
129
130Then, consumers are used to extract data from tracing buffers. These consumers
131are daemon consuming either UST or/and LTTng buffers. For a single session,
132only one UST consumer and one LTTng consumer is necessary. The daemon CAN
133handle multiple tracing buffers for network streaming by example or for quick
134snapshot. These consumers are told by the inprocess library or the kernel to
135start getting out data on disk or network.
136
137For the next subsections, every components of this new proposal is explained
138from the global and per-user registry perspective.
139
140LTT-SESSIOND:
141
142The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to
143all active session and, by active, it means a session in any state other than
144destroyed. Each entity we are keeping track of, here session, will have a
145universal unique identifier (UUID) assigned to it. The purpose of this UUID is
146to track a session in order to apply any kind of actions (Ex: Attach, Destroy).
147A human readable version SHOULD be consider in order to facilitate the session
148identification when listed by lttng.
149
150The daemon creates two local Unix sockets (AF_UNIX). The first one is for what
151we call client communication i.e. interaction with lttng (or any other
152compatible tools). That socket is set with the ltt-sessiond credentials with
153read-write mode for both user and group. The second one is a global socket for
154application registration for the UST case (see inprocess lib subsection below).
155
156This daemon is also responsible for tracing buffers creation. Two main reasons
157motivate this design:
158
159 * The ltt-sessiond needs to keep track of all the shared memory segments in
160 order to be able to give reference to any other possible consumer.
161
162 * For the case of sharing tracing buffers between all userspace
163 applications, having the registry allocating them will allow that but, if
164 the inprocess library was allocating them, we will need to redesign the
165 whole model.
166
167For all tracing actions either to interact with a session or a specific trace,
168the lttng client MUST go through ltt-sessiond. The daemon will take care of
169routing the command to the write inprocess library or the kernel.
170
171Global registry:
172
173A global registry SHOULD be started, idealy at boot, with credentials UID root
174and GID of the tracing group. Only user within the tracing group will be able
175to interact with that registry. All applications will try to register to that
176registry using the global socket (second one discuss above).
177
178Per-user registry:
179
180This type of registry address two use cases. The first one is when a session
181creation is requested from lttng but no global ltt-sessiond exist. So, a
182ltt-sessiond will be spawned in order to manage the tracing of that user. The
183second use case is when a user is not in the tracing group thus he cannot
184communication with the global registry.
185
186However, care MUST be put in order to manage the socket's daemon. They are not
187global anymore so they should be created in the home directory of the user
188requesting tracing.
189
190In both cases, for global and per-user registry, all applications MUST try to
191register to both ltt-sessiond. (see inprocess library subsection for details)
192
193The trace roles of ltt-sessiond:
194
195 Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
196
197 Registry - keep track of trace's information:
198 * shared memory location (only the keyid)
199 * application PID (UST)
200 * type (kernel or UST)
201 * session name
202 * UID
203
204 Buffers creation - creates shared memory for the tracing buffers.
205
206UST-CONSUMERD:
207
208The purpose of this daemon is to consume the UST trace buffers for only a
209specific session. The session MAY have several traces for example two different
210applications. The client tool, lttng has to create the ust-consumerd if NONE
211is available for that session. It is very important to understand that for a
212tracing session, there is only one ust-consumerd for all the traced
213applications.
214
215This daemon basically empty the tracing buffers when asked for and writes that
216data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring
217Frameworks). The inprocess library is the one that tells the ust-consumerd
218daemon that the buffers are ready for consumption.
219
220Here is a flow of action to illustrate the ust-consumerd life span:
221
fa8f9c82 2221)
508f6562 223+-----------+ ops +--------------+
fa8f9c82 224| lttng A |<---------->| ltt-sessiond |
508f6562
DG
225+-----------+ +--------------+
226
227lttng ask for tracing an application using the PID and the session UUID. The
228shared memory reference is given to lttng and the ust-consumerd communication
229socket if ust-consumerd already exist.
230
2312a) If ust-consumerd EXIST
232
fa8f9c82
DG
233+-----------+
234| lttng A |
235+-----------+
236 | mem ref.
237 | +---------------+ read +------------+
508f6562
DG
238 +-->| ust-consumerd |--------->| shared mem |
239 +---------------+ +------------+
240
241In that case, lttng only ask ust-consumerd to consume the buffers using
242the reference it previously got from ltt-sessiond.
243
2442b) If ust-consumerd DOES NOT EXIST
245
246+-----------+ +--------------+
fa8f9c82 247| lttng A | +---->| ltt-sessiond |
508f6562
DG
248+-----------+ | +--------------+
249 | ID |
250 | mem ref. | register
251 | +---------------+
252 +-->| ust-consumerd |
253 +---------------+
254
255lttng spawns the ust-consumerd for the session using the session UUID in
256order for the daemon to register as a consumer to ltt-sessiond for that
257session.
258
259Quick buffer snapshot:
260
2611) Here, lttng will request a buffer snapshot for an already running session.
262
fa8f9c82
DG
263+-----------+ +--------------+
264| lttng A |-------- ops ------->| ltt-sessiond |
265+-----------+ +--------------+
508f6562
DG
266 | | command
267 | +-----------------+ +-------+<--+
268 | | ust-consumerd 1 |<----| app_1 |-+
269 | +-----------------+ +-------+ | write
fa8f9c82 270 | 1 | v
508f6562
DG
271 | | +-------------+
272 | +--- read ----->| shared mem. |
fa8f9c82
DG
273 | +-------------+
274 | ^
508f6562
DG
275 | +-----------------+ |
276 +->| ust-consumerd 2 |----------+
277 +-----------------+ snapshot
278 | write
279 |
280 +---> disk/network
281
282The first ust-consumerd (1) was already consuming buffers for the current
283session. So, lttng ask for a live snapshot. A new ust-consumerd (2) is
284spawned, snapshot the buffers using the shared memory reference from
285ltt-sessiond, writes date to disk and die after all.
286
287On the security side, the ust-consumerd gets UID/GID from the lttng
288credentials since it was spawned by lttng and so the files containing the
289tracing data will also be set to UID/GID of the lttng client. No setuid or
290setgid is used, we only use the credentials of the user.
291
292The roles of ust-consumerd:
fa8f9c82 293
508f6562
DG
294 Register to ltt-sessiond - Using a session UUID and credentials (UID/GID)
295
296 Consume buffers - Write data to a file descriptor (on disk, network, ...)
297
298Buffer consumption is triggered by the inprocess library which tells
299ust-consumerd when to consume.
300
301LTT-CONSUMERD:
302
303The purpose of this daemon is to consume the LTTng trace buffers for only a
fa8f9c82 304specific session.
508f6562
DG
305
306For that kernel consumer, ltt-sessiond will pass different anonymous file
307descriptors to the ltt-consumerd using a Unix socket. From these file
308desriptors, it will be able to get the data from a special function export by
309the LTTng kernel.
310
311ltt-consumerd will be managed by the exact same way as ust-consumerd. However,
312in order to trace the kernel, you are either root (UID=0) or in the tracing
313group.
314
315The roles of ltt-consumerd:
316
317 Register to ltt-sessiond - Using a session UUID and credentials (UID/GID)
318
319 Consume buffers - Write data to a file descriptor (on disk, network, ...)
320
321Kernel triggers ltt-consumerd for buffer consumption.
322
323UST INPROCESS LIBRARY:
324
325When the application starts, this library will check for the global named pipe
326of ltt-sessiond. If present, it MUST validate that root is the owner. This
327check is very important to prevent ltt-sessiond spoofing. If the pipe is root,
328we are certain that it's the privileged user that operates tracing. Then, using
329it's UID, the application will try to register to the per-user ltt-sessiond
330again verifying before the owner ship of the named pipe that should match the
331UID.
332
333Before registration, the inprocess library MUST validate with the ltt-sessiond
334the library version for compatibility reason. This is mechanism is useful for
335library compatibility but also to see if ltt-sessiond socket is valid (means
336that an actual ltt-sessiond is listening on the other side). Having no response
337for over 10 seconds, the application will cut communication on that socket and
338fallback to the application common named pipe (explain below).
339
340If the socket is valid, it will register as a traceable application using the
341apps credentials and will open a local Unix socket, passed to ltt-sessiond, in
342order to receive an eventual shared memory reference. It will then wait on it
343if any other command are given by the lttng client. This socket becomes the
344only channel of communication between the registry and the application.
345
346If no ltt-sessiond is present at registration, the application tries to open
347the application common named pipe or create it if it does not exist and wait on
348it (using poll or epoll Linux API). Having any type of event on that pipe, the
349inprocess library will then try to register to the global and per-user
350ltt-sessiond. If it fails again, it goes back again to wait on that pipe.
351
352SHARED MEMORY
353
354For UST, this is the memory area where the tracing buffers will be held and
355given access in read-write mode for the inprocess library of the application.
356
357On the LTTng side (for ltt-consumerd), these buffers are in the kernel space
358and given access by opening a file in the debugfs file system. With an
359anonymous file desriptor, this consumer will be able to extract the data.
360
361This memory is ONLY used for the tracing data. No communication between
362components is done using that memory.
363
364A shared memory segment for tracing MUST be set with the tracing group GID for
365the UST buffers. This is the job of ltt-sessiond.
366
367PREREQUISITES:
368
369The global ltt-sessiond daemon MUST always be running as "root" or an
370equivalent user having the same privilege as root (UID = 0).
371
372The ltt-sessiond daemon SHOULD be up and running at all time in order to trace
373a tracable application.
374
375The new lttng library API MUST be used to interact with the
fa8f9c82 376ltt-sessiond registry daemon for every trace action needed by the user.
508f6562
DG
377
378A tracing group MUST be created. Whoever is in that group is able to access the
379tracing data of any buffers and is able to trace any application or the kernel.
380
381WARNING: The tracing group name COULD interfere with other already existing
382groups. Care should be put at install time for that (from source and packages)
383
384The next section illustrates different use cases using that new model.
385
386Use Cases
387-----------------
388
389Each case considers these :
390
391* user A - UID: A; GID: A, tracing
392* user B - UID: B; GID: B, tracing
393
394Scenario 1 - Single user tracing app_1
395------
396
397This first scenario shows how user A will start a trace for application app_1
398that is not running.
399
4001) lttng ask ltt-sessiond for a new session through a Unix socket. If
401allowed, ltt-sessiond returns a session UUID to the client.
402(Ex: ops --> new session)
403
404+-----------+ ops +--------------+
405| lttng A |<---------->| ltt-sessiond |
406+-----------+ +--------------+
407
4082) The app_1 is spawned by lttng having the user A credentials. Then, app_1
409automatically register to ltt-sessiond has a "tracable apps" through the global
410named pipe of ltt-sessiond using the UID/GID and session UUID.
411
412The shared memory is created with the app_1 UID (rw-) and tracing group GID
413(r--) and a reference is given back to app_1
414
415+-----------+ +--------------+
416| lttng A | | ltt-sessiond |
417+-----------+ +--------------+
418 | ^ |
419 | +-------+ | | +-------------+
420 +-->| app_1 |<--------+ +-->| shared mem. |
421 +-------+ +-------------+
422
4233) app_1 connect to the shared memory and ust-consumerd is spawned with the
424session UUID and lttng credentials (user A). It then register to ltt-sessiond
425for a valid session to consume using the previous session UUID and credentials.
426
427+-----------+ +--------------+
428| lttng A | +-->| ltt-sessiond |----------+
429+-----------+ | +--------------+ |
430 | | |
431 | +---------------+ read | commands
432 +-->| ust-consumerd |---------+ | and
433 +---------------+ v | options
434 ^ | +-------------+ |
435 | v +------>| shared mem. | |
436 +-------+ | +-------------+ |
437 | app_1 |-------- |
438 +-------+ write |
439 ^ |
440 +---------------------------------------
441
442Scenario 2 - Single user tracing already running app_1
443------
444
4451) lttng ask ltt-sessiond for a new session through a Unix socket. If allowed
446(able to write on socket), ltt-sessiond returns a session UUID to the client.
447
448+-----------+ ops +--------------+
449| lttng A |<---------->| ltt-sessiond |
450+-----------+ +--------------+
fa8f9c82 451 ^
508f6562
DG
452 +-------+ read |
453 | app_1 |----------+
454 +-------+
455
456NOTE: At this stage, since app_1 is already running, the registration of app_1
457to ltt-sessiond has already been done. However, the shared memory segment is
458not allocated yet until a trace session is initiated. Having no shared memory,
459the inprocess library of app_1 will wait on the local Unix socket connected to
460ltt-sessiond for the reference.
461
462+-----------+ +--------------+
463| lttng A | | ltt-sessiond |
464+-----------+ +--------------+
465 ^ |
466 +-------+ | | +-------------+
467 | app_1 |<--------+ +-->| shared mem. |
468 +-------+ commands +-------------+
469 | ^
470 +---------- write ----------+
471
4722) lttng spawns a ust-consumerd for the session. We get the same figure as
473step 3 in the first scenario.
474
475There is a small difference though. The application MAY NOT be using the same
476credentials as user A (lttng). However, the shared memory is always GID of
477the tracing group. So, in order for user A to trace app_1, is MUST be in the
478tracing group otherwise, if the application is not set with the user
479credentials, user A will not be able to trace app_1
480
481Scenario 3 - Multiple users tracing the same running application
482------
483
4841) Session are created for the two users. Using the same exact mechanism as
485before, the shared memory and consumers are created. Two users, two sessions,
486two consumers and two shared memories for the same application.
487
fa8f9c82
DG
488+-----------+ +--------------+
489| lttng A |-------- ops ------->| ltt-sessiond |
490+-----------+ ^ +--------------+
508f6562
DG
491 | ^ commands
492+-----------+ | +-------+<--+
493| lttng B |------+ +--->| app_1 |------- write -----+
fa8f9c82
DG
494+-----------+ | +-------+ |
495 | |
508f6562
DG
496 +-----------------+ | +-------------+ |
497 | ust-consumerd A |--O--- read ----->| shared mem. |<-+
fa8f9c82
DG
498 +-----------------+ | +-------------+ |
499 | |
508f6562
DG
500 +-----------------+ v +-------------+ |
501 | ust-consumerd B |--+--- read ----->| shared mem. |<-+
fa8f9c82 502 +-----------------+ +-------------+
508f6562
DG
503
504ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
505ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
506
507Scenario 4 - User not in the tracing group
508------
509
510For this particular case, it's all goes back to the first scenario. The user
511MUST start the application using his credentials. The session will be created
512by the per-user ltt-sessiond but he will not be able to trace anything that the
513user does not owned.
This page took 0.067848 seconds and 4 git commands to generate.