Commit | Line | Data |
---|---|---|
fa8f9c82 DG |
1 | RFC - LTTng session daemon architecture |
2 | ||
3 | WARNING: | |
9c6bda17 | 4 | Parts of the proposal are obsolete but we keep this version for historical |
fa8f9c82 | 5 | purposes. |
508f6562 DG |
6 | |
7 | Author: David Goulet <david.goulet@polymtl.ca> | |
8 | ||
9 | Contributors: | |
10 | * Mathieu Desnoyers <mathieu.desnoyers@efficios.com> | |
11 | * Yannick Brosseau <yannick.brosseau@polymtl.ca> | |
12 | * Nils Carlson <nils.carlson@ericsson.com> | |
13 | * Michel Dagenais <michel.dagenais@polymtl.ca> | |
14 | * Stefan Hajnoczi <stefanha@gmail.com> | |
15 | ||
16 | Version: | |
17 | - v0.1: 17/01/2011 | |
18 | * Initial proposal | |
19 | ||
fa8f9c82 | 20 | - v0.2: 19/01/2011 |
508f6562 DG |
21 | After multiple reply from all the contributors above, here is the list |
22 | of what has changed: | |
23 | * Change/Add Terminology elements from the initial model | |
24 | * New figures for four new scenarios | |
25 | * Add inprocess library section | |
26 | * LTTng kernel tracer support proposition | |
27 | * More details for the Model and Components | |
28 | * Improve the basic model. Quite different from the last one | |
fa8f9c82 | 29 | |
508f6562 DG |
30 | - v0.3: 28/01/2011 |
31 | In response from Michel Dagenais and Nils Carlson comments: | |
32 | * Add scaling to reasons of this re-engineering | |
33 | * Purpose of the session ID | |
34 | * Explain why ltt-sessiond creates the tracing buffers | |
35 | * ust-consumerd interaction schema | |
36 | * Clarify inprocess library behavior | |
37 | ||
38 | - v0.4: 01/02/2011 | |
39 | After Mathieu Desnoyers and Michel Dagenais comments: | |
40 | * Add section Introduction | |
fa8f9c82 | 41 | * Define the global and per-user ltt-sessiond |
508f6562 DG |
42 | * Add details for ltt-sessiond in the inprocess lib section |
43 | * Session ID are now UUID | |
44 | * Add buffer snapshot schema for ust-consumerd | |
45 | * ltt-sessiond validate inprocess lib version | |
46 | * ltt-sessiond socket validation by the inprocess lib. | |
47 | * Add lttng definition | |
48 | * Add consumer to the Model section | |
fa8f9c82 DG |
49 | - v0.5: 23/07/2012 |
50 | * Please refer to the thesis of David Goulet for the complete | |
51 | and up to date specifications of the architecture and algorithms used. | |
52 | (http://publications.polymtl.ca/842/1/2012_DavidGoulet.pdf) | |
508f6562 DG |
53 | |
54 | Terminology | |
55 | ----------------- | |
56 | ||
57 | ltt-sessiond - Main daemon for trace session registry for UST and LTTng | |
32258573 | 58 | NOTE: Changed to lttng-sessiond in the git tree |
508f6562 DG |
59 | |
60 | ust-consumerd - Daemon that consume UST buffers for a specific application | |
61 | ||
62 | ltt-consumerd - Daemon that consume LTTng buffers | |
63 | ||
64 | tracing session - A trace linked to a set of specific tracepoints and to a set | |
65 | of tracing buffers | |
66 | ||
67 | tracing buffers - Buffers containing tracing data | |
68 | ||
69 | tracing data - Data created by tracing an application | |
70 | ||
71 | inprocess library - UST library linked with the application | |
72 | ||
73 | shared memory - system V shared memory | |
74 | ||
75 | application common named pipe - Global named pipe that triggers application | |
76 | registration, on pipe event, to ltt-sessiond | |
77 | ||
78 | lttng - New command line tool for LTTng and UST tracing control | |
79 | ||
80 | Introduction | |
81 | ----------------- | |
82 | ||
83 | This RFC propose a brand new UST and LTTng daemon model. This re-engineering | |
84 | was mostly driven by the need of: | |
85 | ||
86 | * Better security in terms of access rights on tracing data | |
87 | * Manage tracing session | |
88 | * Scaling in terms of thread/processes needed to perform tracing | |
89 | * LTTng and UST integration in terms of merging traces and session control | |
90 | * Networking such as streaming and remote control over different traces | |
91 | ||
92 | The new model follows the basic principles of having a session registry | |
93 | (ltt-sessiond) and consumers for each tracing session (ust-consumerd and | |
94 | ltt-consumerd). | |
95 | ||
96 | With this proposal, LTTng and UST will share the same tracing session, be | |
97 | managed by the same tool and bring a complete integration between these two | |
98 | powerful tools. | |
99 | ||
100 | NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa. | |
101 | ||
102 | Model | |
103 | ----------------- | |
104 | ||
105 | A global and/or per-user registry keeps track of all tracing sessions. Any user | |
106 | that wants to manage either a kernel trace using LTTng or an application trace | |
107 | with UST must interact with that registry for any possible actions. | |
108 | ||
109 | The model address multiple tracing use cases based on the fact that we | |
110 | introduce a tracing Unix group (tracing group). Only users in that group or | |
111 | root can use the global registry. Other users will create a local registry | |
112 | (per-user registry) that will be completely independent from the global one. | |
113 | ||
114 | Two cases: | |
115 | ||
116 | 1) Users in the tracing group, it's tracing session can consume all tracing | |
117 | buffers from all applications and the kernel. | |
118 | ||
119 | 2) Users NOT in the tracing group, it's tracing session can only consume | |
120 | data from its own applications' buffers hence tracing his applications. | |
121 | ||
122 | A session stored by the registry consist of: | |
123 | ||
124 | * Session name (given by the user or automatically assigned) | |
125 | * List of traces (LTTng or UST) | |
126 | * Tracepoints/markers associated to a trace of that session | |
127 | * UUID | |
128 | * Associated user (UID) | |
129 | ||
130 | Then, consumers are used to extract data from tracing buffers. These consumers | |
131 | are daemon consuming either UST or/and LTTng buffers. For a single session, | |
132 | only one UST consumer and one LTTng consumer is necessary. The daemon CAN | |
133 | handle multiple tracing buffers for network streaming by example or for quick | |
134 | snapshot. These consumers are told by the inprocess library or the kernel to | |
135 | start getting out data on disk or network. | |
136 | ||
137 | For the next subsections, every components of this new proposal is explained | |
138 | from the global and per-user registry perspective. | |
139 | ||
140 | LTT-SESSIOND: | |
141 | ||
142 | The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to | |
143 | all active session and, by active, it means a session in any state other than | |
144 | destroyed. Each entity we are keeping track of, here session, will have a | |
145 | universal unique identifier (UUID) assigned to it. The purpose of this UUID is | |
146 | to track a session in order to apply any kind of actions (Ex: Attach, Destroy). | |
147 | A human readable version SHOULD be consider in order to facilitate the session | |
148 | identification when listed by lttng. | |
149 | ||
150 | The daemon creates two local Unix sockets (AF_UNIX). The first one is for what | |
151 | we call client communication i.e. interaction with lttng (or any other | |
152 | compatible tools). That socket is set with the ltt-sessiond credentials with | |
153 | read-write mode for both user and group. The second one is a global socket for | |
154 | application registration for the UST case (see inprocess lib subsection below). | |
155 | ||
156 | This daemon is also responsible for tracing buffers creation. Two main reasons | |
157 | motivate this design: | |
158 | ||
159 | * The ltt-sessiond needs to keep track of all the shared memory segments in | |
160 | order to be able to give reference to any other possible consumer. | |
161 | ||
162 | * For the case of sharing tracing buffers between all userspace | |
163 | applications, having the registry allocating them will allow that but, if | |
164 | the inprocess library was allocating them, we will need to redesign the | |
165 | whole model. | |
166 | ||
167 | For all tracing actions either to interact with a session or a specific trace, | |
168 | the lttng client MUST go through ltt-sessiond. The daemon will take care of | |
169 | routing the command to the write inprocess library or the kernel. | |
170 | ||
171 | Global registry: | |
172 | ||
173 | A global registry SHOULD be started, idealy at boot, with credentials UID root | |
174 | and GID of the tracing group. Only user within the tracing group will be able | |
175 | to interact with that registry. All applications will try to register to that | |
176 | registry using the global socket (second one discuss above). | |
177 | ||
178 | Per-user registry: | |
179 | ||
180 | This type of registry address two use cases. The first one is when a session | |
181 | creation is requested from lttng but no global ltt-sessiond exist. So, a | |
182 | ltt-sessiond will be spawned in order to manage the tracing of that user. The | |
183 | second use case is when a user is not in the tracing group thus he cannot | |
184 | communication with the global registry. | |
185 | ||
186 | However, care MUST be put in order to manage the socket's daemon. They are not | |
187 | global anymore so they should be created in the home directory of the user | |
188 | requesting tracing. | |
189 | ||
190 | In both cases, for global and per-user registry, all applications MUST try to | |
191 | register to both ltt-sessiond. (see inprocess library subsection for details) | |
192 | ||
193 | The trace roles of ltt-sessiond: | |
194 | ||
195 | Trace interaction - Create, Destroy, Pause, Stop, Start, Set options | |
196 | ||
197 | Registry - keep track of trace's information: | |
198 | * shared memory location (only the keyid) | |
199 | * application PID (UST) | |
200 | * type (kernel or UST) | |
201 | * session name | |
202 | * UID | |
203 | ||
204 | Buffers creation - creates shared memory for the tracing buffers. | |
205 | ||
206 | UST-CONSUMERD: | |
207 | ||
208 | The purpose of this daemon is to consume the UST trace buffers for only a | |
209 | specific session. The session MAY have several traces for example two different | |
210 | applications. The client tool, lttng has to create the ust-consumerd if NONE | |
211 | is available for that session. It is very important to understand that for a | |
212 | tracing session, there is only one ust-consumerd for all the traced | |
213 | applications. | |
214 | ||
215 | This daemon basically empty the tracing buffers when asked for and writes that | |
216 | data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring | |
217 | Frameworks). The inprocess library is the one that tells the ust-consumerd | |
218 | daemon that the buffers are ready for consumption. | |
219 | ||
220 | Here is a flow of action to illustrate the ust-consumerd life span: | |
221 | ||
fa8f9c82 | 222 | 1) |
508f6562 | 223 | +-----------+ ops +--------------+ |
fa8f9c82 | 224 | | lttng A |<---------->| ltt-sessiond | |
508f6562 DG |
225 | +-----------+ +--------------+ |
226 | ||
227 | lttng ask for tracing an application using the PID and the session UUID. The | |
228 | shared memory reference is given to lttng and the ust-consumerd communication | |
229 | socket if ust-consumerd already exist. | |
230 | ||
231 | 2a) If ust-consumerd EXIST | |
232 | ||
fa8f9c82 DG |
233 | +-----------+ |
234 | | lttng A | | |
235 | +-----------+ | |
236 | | mem ref. | |
237 | | +---------------+ read +------------+ | |
508f6562 DG |
238 | +-->| ust-consumerd |--------->| shared mem | |
239 | +---------------+ +------------+ | |
240 | ||
241 | In that case, lttng only ask ust-consumerd to consume the buffers using | |
242 | the reference it previously got from ltt-sessiond. | |
243 | ||
244 | 2b) If ust-consumerd DOES NOT EXIST | |
245 | ||
246 | +-----------+ +--------------+ | |
fa8f9c82 | 247 | | lttng A | +---->| ltt-sessiond | |
508f6562 DG |
248 | +-----------+ | +--------------+ |
249 | | ID | | |
250 | | mem ref. | register | |
251 | | +---------------+ | |
252 | +-->| ust-consumerd | | |
253 | +---------------+ | |
254 | ||
255 | lttng spawns the ust-consumerd for the session using the session UUID in | |
256 | order for the daemon to register as a consumer to ltt-sessiond for that | |
257 | session. | |
258 | ||
259 | Quick buffer snapshot: | |
260 | ||
261 | 1) Here, lttng will request a buffer snapshot for an already running session. | |
262 | ||
fa8f9c82 DG |
263 | +-----------+ +--------------+ |
264 | | lttng A |-------- ops ------->| ltt-sessiond | | |
265 | +-----------+ +--------------+ | |
508f6562 DG |
266 | | | command |
267 | | +-----------------+ +-------+<--+ | |
268 | | | ust-consumerd 1 |<----| app_1 |-+ | |
269 | | +-----------------+ +-------+ | write | |
fa8f9c82 | 270 | | 1 | v |
508f6562 DG |
271 | | | +-------------+ |
272 | | +--- read ----->| shared mem. | | |
fa8f9c82 DG |
273 | | +-------------+ |
274 | | ^ | |
508f6562 DG |
275 | | +-----------------+ | |
276 | +->| ust-consumerd 2 |----------+ | |
277 | +-----------------+ snapshot | |
278 | | write | |
279 | | | |
280 | +---> disk/network | |
281 | ||
282 | The first ust-consumerd (1) was already consuming buffers for the current | |
283 | session. So, lttng ask for a live snapshot. A new ust-consumerd (2) is | |
284 | spawned, snapshot the buffers using the shared memory reference from | |
285 | ltt-sessiond, writes date to disk and die after all. | |
286 | ||
287 | On the security side, the ust-consumerd gets UID/GID from the lttng | |
288 | credentials since it was spawned by lttng and so the files containing the | |
289 | tracing data will also be set to UID/GID of the lttng client. No setuid or | |
290 | setgid is used, we only use the credentials of the user. | |
291 | ||
292 | The roles of ust-consumerd: | |
fa8f9c82 | 293 | |
508f6562 DG |
294 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) |
295 | ||
296 | Consume buffers - Write data to a file descriptor (on disk, network, ...) | |
297 | ||
298 | Buffer consumption is triggered by the inprocess library which tells | |
299 | ust-consumerd when to consume. | |
300 | ||
301 | LTT-CONSUMERD: | |
302 | ||
303 | The purpose of this daemon is to consume the LTTng trace buffers for only a | |
fa8f9c82 | 304 | specific session. |
508f6562 DG |
305 | |
306 | For that kernel consumer, ltt-sessiond will pass different anonymous file | |
307 | descriptors to the ltt-consumerd using a Unix socket. From these file | |
308 | desriptors, it will be able to get the data from a special function export by | |
309 | the LTTng kernel. | |
310 | ||
311 | ltt-consumerd will be managed by the exact same way as ust-consumerd. However, | |
312 | in order to trace the kernel, you are either root (UID=0) or in the tracing | |
313 | group. | |
314 | ||
315 | The roles of ltt-consumerd: | |
316 | ||
317 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) | |
318 | ||
319 | Consume buffers - Write data to a file descriptor (on disk, network, ...) | |
320 | ||
321 | Kernel triggers ltt-consumerd for buffer consumption. | |
322 | ||
323 | UST INPROCESS LIBRARY: | |
324 | ||
325 | When the application starts, this library will check for the global named pipe | |
326 | of ltt-sessiond. If present, it MUST validate that root is the owner. This | |
327 | check is very important to prevent ltt-sessiond spoofing. If the pipe is root, | |
328 | we are certain that it's the privileged user that operates tracing. Then, using | |
329 | it's UID, the application will try to register to the per-user ltt-sessiond | |
330 | again verifying before the owner ship of the named pipe that should match the | |
331 | UID. | |
332 | ||
333 | Before registration, the inprocess library MUST validate with the ltt-sessiond | |
334 | the library version for compatibility reason. This is mechanism is useful for | |
335 | library compatibility but also to see if ltt-sessiond socket is valid (means | |
336 | that an actual ltt-sessiond is listening on the other side). Having no response | |
337 | for over 10 seconds, the application will cut communication on that socket and | |
338 | fallback to the application common named pipe (explain below). | |
339 | ||
340 | If the socket is valid, it will register as a traceable application using the | |
341 | apps credentials and will open a local Unix socket, passed to ltt-sessiond, in | |
342 | order to receive an eventual shared memory reference. It will then wait on it | |
343 | if any other command are given by the lttng client. This socket becomes the | |
344 | only channel of communication between the registry and the application. | |
345 | ||
346 | If no ltt-sessiond is present at registration, the application tries to open | |
347 | the application common named pipe or create it if it does not exist and wait on | |
348 | it (using poll or epoll Linux API). Having any type of event on that pipe, the | |
349 | inprocess library will then try to register to the global and per-user | |
350 | ltt-sessiond. If it fails again, it goes back again to wait on that pipe. | |
351 | ||
352 | SHARED MEMORY | |
353 | ||
354 | For UST, this is the memory area where the tracing buffers will be held and | |
355 | given access in read-write mode for the inprocess library of the application. | |
356 | ||
357 | On the LTTng side (for ltt-consumerd), these buffers are in the kernel space | |
358 | and given access by opening a file in the debugfs file system. With an | |
359 | anonymous file desriptor, this consumer will be able to extract the data. | |
360 | ||
361 | This memory is ONLY used for the tracing data. No communication between | |
362 | components is done using that memory. | |
363 | ||
364 | A shared memory segment for tracing MUST be set with the tracing group GID for | |
365 | the UST buffers. This is the job of ltt-sessiond. | |
366 | ||
367 | PREREQUISITES: | |
368 | ||
369 | The global ltt-sessiond daemon MUST always be running as "root" or an | |
370 | equivalent user having the same privilege as root (UID = 0). | |
371 | ||
372 | The ltt-sessiond daemon SHOULD be up and running at all time in order to trace | |
373 | a tracable application. | |
374 | ||
375 | The new lttng library API MUST be used to interact with the | |
fa8f9c82 | 376 | ltt-sessiond registry daemon for every trace action needed by the user. |
508f6562 DG |
377 | |
378 | A tracing group MUST be created. Whoever is in that group is able to access the | |
379 | tracing data of any buffers and is able to trace any application or the kernel. | |
380 | ||
381 | WARNING: The tracing group name COULD interfere with other already existing | |
382 | groups. Care should be put at install time for that (from source and packages) | |
383 | ||
384 | The next section illustrates different use cases using that new model. | |
385 | ||
386 | Use Cases | |
387 | ----------------- | |
388 | ||
389 | Each case considers these : | |
390 | ||
391 | * user A - UID: A; GID: A, tracing | |
392 | * user B - UID: B; GID: B, tracing | |
393 | ||
394 | Scenario 1 - Single user tracing app_1 | |
395 | ------ | |
396 | ||
397 | This first scenario shows how user A will start a trace for application app_1 | |
398 | that is not running. | |
399 | ||
400 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If | |
401 | allowed, ltt-sessiond returns a session UUID to the client. | |
402 | (Ex: ops --> new session) | |
403 | ||
404 | +-----------+ ops +--------------+ | |
405 | | lttng A |<---------->| ltt-sessiond | | |
406 | +-----------+ +--------------+ | |
407 | ||
408 | 2) The app_1 is spawned by lttng having the user A credentials. Then, app_1 | |
409 | automatically register to ltt-sessiond has a "tracable apps" through the global | |
410 | named pipe of ltt-sessiond using the UID/GID and session UUID. | |
411 | ||
412 | The shared memory is created with the app_1 UID (rw-) and tracing group GID | |
413 | (r--) and a reference is given back to app_1 | |
414 | ||
415 | +-----------+ +--------------+ | |
416 | | lttng A | | ltt-sessiond | | |
417 | +-----------+ +--------------+ | |
418 | | ^ | | |
419 | | +-------+ | | +-------------+ | |
420 | +-->| app_1 |<--------+ +-->| shared mem. | | |
421 | +-------+ +-------------+ | |
422 | ||
423 | 3) app_1 connect to the shared memory and ust-consumerd is spawned with the | |
424 | session UUID and lttng credentials (user A). It then register to ltt-sessiond | |
425 | for a valid session to consume using the previous session UUID and credentials. | |
426 | ||
427 | +-----------+ +--------------+ | |
428 | | lttng A | +-->| ltt-sessiond |----------+ | |
429 | +-----------+ | +--------------+ | | |
430 | | | | | |
431 | | +---------------+ read | commands | |
432 | +-->| ust-consumerd |---------+ | and | |
433 | +---------------+ v | options | |
434 | ^ | +-------------+ | | |
435 | | v +------>| shared mem. | | | |
436 | +-------+ | +-------------+ | | |
437 | | app_1 |-------- | | |
438 | +-------+ write | | |
439 | ^ | | |
440 | +--------------------------------------- | |
441 | ||
442 | Scenario 2 - Single user tracing already running app_1 | |
443 | ------ | |
444 | ||
445 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If allowed | |
446 | (able to write on socket), ltt-sessiond returns a session UUID to the client. | |
447 | ||
448 | +-----------+ ops +--------------+ | |
449 | | lttng A |<---------->| ltt-sessiond | | |
450 | +-----------+ +--------------+ | |
fa8f9c82 | 451 | ^ |
508f6562 DG |
452 | +-------+ read | |
453 | | app_1 |----------+ | |
454 | +-------+ | |
455 | ||
456 | NOTE: At this stage, since app_1 is already running, the registration of app_1 | |
457 | to ltt-sessiond has already been done. However, the shared memory segment is | |
458 | not allocated yet until a trace session is initiated. Having no shared memory, | |
459 | the inprocess library of app_1 will wait on the local Unix socket connected to | |
460 | ltt-sessiond for the reference. | |
461 | ||
462 | +-----------+ +--------------+ | |
463 | | lttng A | | ltt-sessiond | | |
464 | +-----------+ +--------------+ | |
465 | ^ | | |
466 | +-------+ | | +-------------+ | |
467 | | app_1 |<--------+ +-->| shared mem. | | |
468 | +-------+ commands +-------------+ | |
469 | | ^ | |
470 | +---------- write ----------+ | |
471 | ||
472 | 2) lttng spawns a ust-consumerd for the session. We get the same figure as | |
473 | step 3 in the first scenario. | |
474 | ||
475 | There is a small difference though. The application MAY NOT be using the same | |
476 | credentials as user A (lttng). However, the shared memory is always GID of | |
477 | the tracing group. So, in order for user A to trace app_1, is MUST be in the | |
478 | tracing group otherwise, if the application is not set with the user | |
479 | credentials, user A will not be able to trace app_1 | |
480 | ||
481 | Scenario 3 - Multiple users tracing the same running application | |
482 | ------ | |
483 | ||
484 | 1) Session are created for the two users. Using the same exact mechanism as | |
485 | before, the shared memory and consumers are created. Two users, two sessions, | |
486 | two consumers and two shared memories for the same application. | |
487 | ||
fa8f9c82 DG |
488 | +-----------+ +--------------+ |
489 | | lttng A |-------- ops ------->| ltt-sessiond | | |
490 | +-----------+ ^ +--------------+ | |
508f6562 DG |
491 | | ^ commands |
492 | +-----------+ | +-------+<--+ | |
493 | | lttng B |------+ +--->| app_1 |------- write -----+ | |
fa8f9c82 DG |
494 | +-----------+ | +-------+ | |
495 | | | | |
508f6562 DG |
496 | +-----------------+ | +-------------+ | |
497 | | ust-consumerd A |--O--- read ----->| shared mem. |<-+ | |
fa8f9c82 DG |
498 | +-----------------+ | +-------------+ | |
499 | | | | |
508f6562 DG |
500 | +-----------------+ v +-------------+ | |
501 | | ust-consumerd B |--+--- read ----->| shared mem. |<-+ | |
fa8f9c82 | 502 | +-----------------+ +-------------+ |
508f6562 DG |
503 | |
504 | ust-consumerd A - UID: user A (rw-), GID: tracing (r--) | |
505 | ust-consumerd B - UID: user B (rw-), GID: tracing (r--) | |
506 | ||
507 | Scenario 4 - User not in the tracing group | |
508 | ------ | |
509 | ||
510 | For this particular case, it's all goes back to the first scenario. The user | |
511 | MUST start the application using his credentials. The session will be created | |
512 | by the per-user ltt-sessiond but he will not be able to trace anything that the | |
513 | user does not owned. |