Commit | Line | Data |
---|---|---|
508f6562 DG |
1 | RFC - New processes model for UST and LTTng |
2 | ||
3 | Author: David Goulet <david.goulet@polymtl.ca> | |
4 | ||
5 | Contributors: | |
6 | * Mathieu Desnoyers <mathieu.desnoyers@efficios.com> | |
7 | * Yannick Brosseau <yannick.brosseau@polymtl.ca> | |
8 | * Nils Carlson <nils.carlson@ericsson.com> | |
9 | * Michel Dagenais <michel.dagenais@polymtl.ca> | |
10 | * Stefan Hajnoczi <stefanha@gmail.com> | |
11 | ||
12 | Version: | |
13 | - v0.1: 17/01/2011 | |
14 | * Initial proposal | |
15 | ||
16 | - v0.2: 19/01/2011 | |
17 | After multiple reply from all the contributors above, here is the list | |
18 | of what has changed: | |
19 | * Change/Add Terminology elements from the initial model | |
20 | * New figures for four new scenarios | |
21 | * Add inprocess library section | |
22 | * LTTng kernel tracer support proposition | |
23 | * More details for the Model and Components | |
24 | * Improve the basic model. Quite different from the last one | |
25 | ||
26 | - v0.3: 28/01/2011 | |
27 | In response from Michel Dagenais and Nils Carlson comments: | |
28 | * Add scaling to reasons of this re-engineering | |
29 | * Purpose of the session ID | |
30 | * Explain why ltt-sessiond creates the tracing buffers | |
31 | * ust-consumerd interaction schema | |
32 | * Clarify inprocess library behavior | |
33 | ||
34 | - v0.4: 01/02/2011 | |
35 | After Mathieu Desnoyers and Michel Dagenais comments: | |
36 | * Add section Introduction | |
37 | * Define the global and per-user ltt-sessiond | |
38 | * Add details for ltt-sessiond in the inprocess lib section | |
39 | * Session ID are now UUID | |
40 | * Add buffer snapshot schema for ust-consumerd | |
41 | * ltt-sessiond validate inprocess lib version | |
42 | * ltt-sessiond socket validation by the inprocess lib. | |
43 | * Add lttng definition | |
44 | * Add consumer to the Model section | |
45 | ||
46 | Terminology | |
47 | ----------------- | |
48 | ||
49 | ltt-sessiond - Main daemon for trace session registry for UST and LTTng | |
50 | ||
51 | ust-consumerd - Daemon that consume UST buffers for a specific application | |
52 | ||
53 | ltt-consumerd - Daemon that consume LTTng buffers | |
54 | ||
55 | tracing session - A trace linked to a set of specific tracepoints and to a set | |
56 | of tracing buffers | |
57 | ||
58 | tracing buffers - Buffers containing tracing data | |
59 | ||
60 | tracing data - Data created by tracing an application | |
61 | ||
62 | inprocess library - UST library linked with the application | |
63 | ||
64 | shared memory - system V shared memory | |
65 | ||
66 | application common named pipe - Global named pipe that triggers application | |
67 | registration, on pipe event, to ltt-sessiond | |
68 | ||
69 | lttng - New command line tool for LTTng and UST tracing control | |
70 | ||
71 | Introduction | |
72 | ----------------- | |
73 | ||
74 | This RFC propose a brand new UST and LTTng daemon model. This re-engineering | |
75 | was mostly driven by the need of: | |
76 | ||
77 | * Better security in terms of access rights on tracing data | |
78 | * Manage tracing session | |
79 | * Scaling in terms of thread/processes needed to perform tracing | |
80 | * LTTng and UST integration in terms of merging traces and session control | |
81 | * Networking such as streaming and remote control over different traces | |
82 | ||
83 | The new model follows the basic principles of having a session registry | |
84 | (ltt-sessiond) and consumers for each tracing session (ust-consumerd and | |
85 | ltt-consumerd). | |
86 | ||
87 | With this proposal, LTTng and UST will share the same tracing session, be | |
88 | managed by the same tool and bring a complete integration between these two | |
89 | powerful tools. | |
90 | ||
91 | NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa. | |
92 | ||
93 | Model | |
94 | ----------------- | |
95 | ||
96 | A global and/or per-user registry keeps track of all tracing sessions. Any user | |
97 | that wants to manage either a kernel trace using LTTng or an application trace | |
98 | with UST must interact with that registry for any possible actions. | |
99 | ||
100 | The model address multiple tracing use cases based on the fact that we | |
101 | introduce a tracing Unix group (tracing group). Only users in that group or | |
102 | root can use the global registry. Other users will create a local registry | |
103 | (per-user registry) that will be completely independent from the global one. | |
104 | ||
105 | Two cases: | |
106 | ||
107 | 1) Users in the tracing group, it's tracing session can consume all tracing | |
108 | buffers from all applications and the kernel. | |
109 | ||
110 | 2) Users NOT in the tracing group, it's tracing session can only consume | |
111 | data from its own applications' buffers hence tracing his applications. | |
112 | ||
113 | A session stored by the registry consist of: | |
114 | ||
115 | * Session name (given by the user or automatically assigned) | |
116 | * List of traces (LTTng or UST) | |
117 | * Tracepoints/markers associated to a trace of that session | |
118 | * UUID | |
119 | * Associated user (UID) | |
120 | ||
121 | Then, consumers are used to extract data from tracing buffers. These consumers | |
122 | are daemon consuming either UST or/and LTTng buffers. For a single session, | |
123 | only one UST consumer and one LTTng consumer is necessary. The daemon CAN | |
124 | handle multiple tracing buffers for network streaming by example or for quick | |
125 | snapshot. These consumers are told by the inprocess library or the kernel to | |
126 | start getting out data on disk or network. | |
127 | ||
128 | For the next subsections, every components of this new proposal is explained | |
129 | from the global and per-user registry perspective. | |
130 | ||
131 | LTT-SESSIOND: | |
132 | ||
133 | The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to | |
134 | all active session and, by active, it means a session in any state other than | |
135 | destroyed. Each entity we are keeping track of, here session, will have a | |
136 | universal unique identifier (UUID) assigned to it. The purpose of this UUID is | |
137 | to track a session in order to apply any kind of actions (Ex: Attach, Destroy). | |
138 | A human readable version SHOULD be consider in order to facilitate the session | |
139 | identification when listed by lttng. | |
140 | ||
141 | The daemon creates two local Unix sockets (AF_UNIX). The first one is for what | |
142 | we call client communication i.e. interaction with lttng (or any other | |
143 | compatible tools). That socket is set with the ltt-sessiond credentials with | |
144 | read-write mode for both user and group. The second one is a global socket for | |
145 | application registration for the UST case (see inprocess lib subsection below). | |
146 | ||
147 | This daemon is also responsible for tracing buffers creation. Two main reasons | |
148 | motivate this design: | |
149 | ||
150 | * The ltt-sessiond needs to keep track of all the shared memory segments in | |
151 | order to be able to give reference to any other possible consumer. | |
152 | ||
153 | * For the case of sharing tracing buffers between all userspace | |
154 | applications, having the registry allocating them will allow that but, if | |
155 | the inprocess library was allocating them, we will need to redesign the | |
156 | whole model. | |
157 | ||
158 | For all tracing actions either to interact with a session or a specific trace, | |
159 | the lttng client MUST go through ltt-sessiond. The daemon will take care of | |
160 | routing the command to the write inprocess library or the kernel. | |
161 | ||
162 | Global registry: | |
163 | ||
164 | A global registry SHOULD be started, idealy at boot, with credentials UID root | |
165 | and GID of the tracing group. Only user within the tracing group will be able | |
166 | to interact with that registry. All applications will try to register to that | |
167 | registry using the global socket (second one discuss above). | |
168 | ||
169 | Per-user registry: | |
170 | ||
171 | This type of registry address two use cases. The first one is when a session | |
172 | creation is requested from lttng but no global ltt-sessiond exist. So, a | |
173 | ltt-sessiond will be spawned in order to manage the tracing of that user. The | |
174 | second use case is when a user is not in the tracing group thus he cannot | |
175 | communication with the global registry. | |
176 | ||
177 | However, care MUST be put in order to manage the socket's daemon. They are not | |
178 | global anymore so they should be created in the home directory of the user | |
179 | requesting tracing. | |
180 | ||
181 | In both cases, for global and per-user registry, all applications MUST try to | |
182 | register to both ltt-sessiond. (see inprocess library subsection for details) | |
183 | ||
184 | The trace roles of ltt-sessiond: | |
185 | ||
186 | Trace interaction - Create, Destroy, Pause, Stop, Start, Set options | |
187 | ||
188 | Registry - keep track of trace's information: | |
189 | * shared memory location (only the keyid) | |
190 | * application PID (UST) | |
191 | * type (kernel or UST) | |
192 | * session name | |
193 | * UID | |
194 | ||
195 | Buffers creation - creates shared memory for the tracing buffers. | |
196 | ||
197 | UST-CONSUMERD: | |
198 | ||
199 | The purpose of this daemon is to consume the UST trace buffers for only a | |
200 | specific session. The session MAY have several traces for example two different | |
201 | applications. The client tool, lttng has to create the ust-consumerd if NONE | |
202 | is available for that session. It is very important to understand that for a | |
203 | tracing session, there is only one ust-consumerd for all the traced | |
204 | applications. | |
205 | ||
206 | This daemon basically empty the tracing buffers when asked for and writes that | |
207 | data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring | |
208 | Frameworks). The inprocess library is the one that tells the ust-consumerd | |
209 | daemon that the buffers are ready for consumption. | |
210 | ||
211 | Here is a flow of action to illustrate the ust-consumerd life span: | |
212 | ||
213 | 1) | |
214 | +-----------+ ops +--------------+ | |
215 | | lttng A |<---------->| ltt-sessiond | | |
216 | +-----------+ +--------------+ | |
217 | ||
218 | lttng ask for tracing an application using the PID and the session UUID. The | |
219 | shared memory reference is given to lttng and the ust-consumerd communication | |
220 | socket if ust-consumerd already exist. | |
221 | ||
222 | 2a) If ust-consumerd EXIST | |
223 | ||
224 | +-----------+ | |
225 | | lttng A | | |
226 | +-----------+ | |
227 | | mem ref. | |
228 | | +---------------+ read +------------+ | |
229 | +-->| ust-consumerd |--------->| shared mem | | |
230 | +---------------+ +------------+ | |
231 | ||
232 | In that case, lttng only ask ust-consumerd to consume the buffers using | |
233 | the reference it previously got from ltt-sessiond. | |
234 | ||
235 | 2b) If ust-consumerd DOES NOT EXIST | |
236 | ||
237 | +-----------+ +--------------+ | |
238 | | lttng A | +---->| ltt-sessiond | | |
239 | +-----------+ | +--------------+ | |
240 | | ID | | |
241 | | mem ref. | register | |
242 | | +---------------+ | |
243 | +-->| ust-consumerd | | |
244 | +---------------+ | |
245 | ||
246 | lttng spawns the ust-consumerd for the session using the session UUID in | |
247 | order for the daemon to register as a consumer to ltt-sessiond for that | |
248 | session. | |
249 | ||
250 | Quick buffer snapshot: | |
251 | ||
252 | 1) Here, lttng will request a buffer snapshot for an already running session. | |
253 | ||
254 | +-----------+ +--------------+ | |
255 | | lttng A |-------- ops ------->| ltt-sessiond | | |
256 | +-----------+ +--------------+ | |
257 | | | command | |
258 | | +-----------------+ +-------+<--+ | |
259 | | | ust-consumerd 1 |<----| app_1 |-+ | |
260 | | +-----------------+ +-------+ | write | |
261 | | 1 | v | |
262 | | | +-------------+ | |
263 | | +--- read ----->| shared mem. | | |
264 | | +-------------+ | |
265 | | ^ | |
266 | | +-----------------+ | | |
267 | +->| ust-consumerd 2 |----------+ | |
268 | +-----------------+ snapshot | |
269 | | write | |
270 | | | |
271 | +---> disk/network | |
272 | ||
273 | The first ust-consumerd (1) was already consuming buffers for the current | |
274 | session. So, lttng ask for a live snapshot. A new ust-consumerd (2) is | |
275 | spawned, snapshot the buffers using the shared memory reference from | |
276 | ltt-sessiond, writes date to disk and die after all. | |
277 | ||
278 | On the security side, the ust-consumerd gets UID/GID from the lttng | |
279 | credentials since it was spawned by lttng and so the files containing the | |
280 | tracing data will also be set to UID/GID of the lttng client. No setuid or | |
281 | setgid is used, we only use the credentials of the user. | |
282 | ||
283 | The roles of ust-consumerd: | |
284 | ||
285 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) | |
286 | ||
287 | Consume buffers - Write data to a file descriptor (on disk, network, ...) | |
288 | ||
289 | Buffer consumption is triggered by the inprocess library which tells | |
290 | ust-consumerd when to consume. | |
291 | ||
292 | LTT-CONSUMERD: | |
293 | ||
294 | The purpose of this daemon is to consume the LTTng trace buffers for only a | |
295 | specific session. | |
296 | ||
297 | For that kernel consumer, ltt-sessiond will pass different anonymous file | |
298 | descriptors to the ltt-consumerd using a Unix socket. From these file | |
299 | desriptors, it will be able to get the data from a special function export by | |
300 | the LTTng kernel. | |
301 | ||
302 | ltt-consumerd will be managed by the exact same way as ust-consumerd. However, | |
303 | in order to trace the kernel, you are either root (UID=0) or in the tracing | |
304 | group. | |
305 | ||
306 | The roles of ltt-consumerd: | |
307 | ||
308 | Register to ltt-sessiond - Using a session UUID and credentials (UID/GID) | |
309 | ||
310 | Consume buffers - Write data to a file descriptor (on disk, network, ...) | |
311 | ||
312 | Kernel triggers ltt-consumerd for buffer consumption. | |
313 | ||
314 | UST INPROCESS LIBRARY: | |
315 | ||
316 | When the application starts, this library will check for the global named pipe | |
317 | of ltt-sessiond. If present, it MUST validate that root is the owner. This | |
318 | check is very important to prevent ltt-sessiond spoofing. If the pipe is root, | |
319 | we are certain that it's the privileged user that operates tracing. Then, using | |
320 | it's UID, the application will try to register to the per-user ltt-sessiond | |
321 | again verifying before the owner ship of the named pipe that should match the | |
322 | UID. | |
323 | ||
324 | Before registration, the inprocess library MUST validate with the ltt-sessiond | |
325 | the library version for compatibility reason. This is mechanism is useful for | |
326 | library compatibility but also to see if ltt-sessiond socket is valid (means | |
327 | that an actual ltt-sessiond is listening on the other side). Having no response | |
328 | for over 10 seconds, the application will cut communication on that socket and | |
329 | fallback to the application common named pipe (explain below). | |
330 | ||
331 | If the socket is valid, it will register as a traceable application using the | |
332 | apps credentials and will open a local Unix socket, passed to ltt-sessiond, in | |
333 | order to receive an eventual shared memory reference. It will then wait on it | |
334 | if any other command are given by the lttng client. This socket becomes the | |
335 | only channel of communication between the registry and the application. | |
336 | ||
337 | If no ltt-sessiond is present at registration, the application tries to open | |
338 | the application common named pipe or create it if it does not exist and wait on | |
339 | it (using poll or epoll Linux API). Having any type of event on that pipe, the | |
340 | inprocess library will then try to register to the global and per-user | |
341 | ltt-sessiond. If it fails again, it goes back again to wait on that pipe. | |
342 | ||
343 | SHARED MEMORY | |
344 | ||
345 | For UST, this is the memory area where the tracing buffers will be held and | |
346 | given access in read-write mode for the inprocess library of the application. | |
347 | ||
348 | On the LTTng side (for ltt-consumerd), these buffers are in the kernel space | |
349 | and given access by opening a file in the debugfs file system. With an | |
350 | anonymous file desriptor, this consumer will be able to extract the data. | |
351 | ||
352 | This memory is ONLY used for the tracing data. No communication between | |
353 | components is done using that memory. | |
354 | ||
355 | A shared memory segment for tracing MUST be set with the tracing group GID for | |
356 | the UST buffers. This is the job of ltt-sessiond. | |
357 | ||
358 | PREREQUISITES: | |
359 | ||
360 | The global ltt-sessiond daemon MUST always be running as "root" or an | |
361 | equivalent user having the same privilege as root (UID = 0). | |
362 | ||
363 | The ltt-sessiond daemon SHOULD be up and running at all time in order to trace | |
364 | a tracable application. | |
365 | ||
366 | The new lttng library API MUST be used to interact with the | |
367 | ltt-sessiond registry daemon for every trace action needed by the user. | |
368 | ||
369 | A tracing group MUST be created. Whoever is in that group is able to access the | |
370 | tracing data of any buffers and is able to trace any application or the kernel. | |
371 | ||
372 | WARNING: The tracing group name COULD interfere with other already existing | |
373 | groups. Care should be put at install time for that (from source and packages) | |
374 | ||
375 | The next section illustrates different use cases using that new model. | |
376 | ||
377 | Use Cases | |
378 | ----------------- | |
379 | ||
380 | Each case considers these : | |
381 | ||
382 | * user A - UID: A; GID: A, tracing | |
383 | * user B - UID: B; GID: B, tracing | |
384 | ||
385 | Scenario 1 - Single user tracing app_1 | |
386 | ------ | |
387 | ||
388 | This first scenario shows how user A will start a trace for application app_1 | |
389 | that is not running. | |
390 | ||
391 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If | |
392 | allowed, ltt-sessiond returns a session UUID to the client. | |
393 | (Ex: ops --> new session) | |
394 | ||
395 | +-----------+ ops +--------------+ | |
396 | | lttng A |<---------->| ltt-sessiond | | |
397 | +-----------+ +--------------+ | |
398 | ||
399 | 2) The app_1 is spawned by lttng having the user A credentials. Then, app_1 | |
400 | automatically register to ltt-sessiond has a "tracable apps" through the global | |
401 | named pipe of ltt-sessiond using the UID/GID and session UUID. | |
402 | ||
403 | The shared memory is created with the app_1 UID (rw-) and tracing group GID | |
404 | (r--) and a reference is given back to app_1 | |
405 | ||
406 | +-----------+ +--------------+ | |
407 | | lttng A | | ltt-sessiond | | |
408 | +-----------+ +--------------+ | |
409 | | ^ | | |
410 | | +-------+ | | +-------------+ | |
411 | +-->| app_1 |<--------+ +-->| shared mem. | | |
412 | +-------+ +-------------+ | |
413 | ||
414 | 3) app_1 connect to the shared memory and ust-consumerd is spawned with the | |
415 | session UUID and lttng credentials (user A). It then register to ltt-sessiond | |
416 | for a valid session to consume using the previous session UUID and credentials. | |
417 | ||
418 | +-----------+ +--------------+ | |
419 | | lttng A | +-->| ltt-sessiond |----------+ | |
420 | +-----------+ | +--------------+ | | |
421 | | | | | |
422 | | +---------------+ read | commands | |
423 | +-->| ust-consumerd |---------+ | and | |
424 | +---------------+ v | options | |
425 | ^ | +-------------+ | | |
426 | | v +------>| shared mem. | | | |
427 | +-------+ | +-------------+ | | |
428 | | app_1 |-------- | | |
429 | +-------+ write | | |
430 | ^ | | |
431 | +--------------------------------------- | |
432 | ||
433 | Scenario 2 - Single user tracing already running app_1 | |
434 | ------ | |
435 | ||
436 | 1) lttng ask ltt-sessiond for a new session through a Unix socket. If allowed | |
437 | (able to write on socket), ltt-sessiond returns a session UUID to the client. | |
438 | ||
439 | +-----------+ ops +--------------+ | |
440 | | lttng A |<---------->| ltt-sessiond | | |
441 | +-----------+ +--------------+ | |
442 | ^ | |
443 | +-------+ read | | |
444 | | app_1 |----------+ | |
445 | +-------+ | |
446 | ||
447 | NOTE: At this stage, since app_1 is already running, the registration of app_1 | |
448 | to ltt-sessiond has already been done. However, the shared memory segment is | |
449 | not allocated yet until a trace session is initiated. Having no shared memory, | |
450 | the inprocess library of app_1 will wait on the local Unix socket connected to | |
451 | ltt-sessiond for the reference. | |
452 | ||
453 | +-----------+ +--------------+ | |
454 | | lttng A | | ltt-sessiond | | |
455 | +-----------+ +--------------+ | |
456 | ^ | | |
457 | +-------+ | | +-------------+ | |
458 | | app_1 |<--------+ +-->| shared mem. | | |
459 | +-------+ commands +-------------+ | |
460 | | ^ | |
461 | +---------- write ----------+ | |
462 | ||
463 | 2) lttng spawns a ust-consumerd for the session. We get the same figure as | |
464 | step 3 in the first scenario. | |
465 | ||
466 | There is a small difference though. The application MAY NOT be using the same | |
467 | credentials as user A (lttng). However, the shared memory is always GID of | |
468 | the tracing group. So, in order for user A to trace app_1, is MUST be in the | |
469 | tracing group otherwise, if the application is not set with the user | |
470 | credentials, user A will not be able to trace app_1 | |
471 | ||
472 | Scenario 3 - Multiple users tracing the same running application | |
473 | ------ | |
474 | ||
475 | 1) Session are created for the two users. Using the same exact mechanism as | |
476 | before, the shared memory and consumers are created. Two users, two sessions, | |
477 | two consumers and two shared memories for the same application. | |
478 | ||
479 | +-----------+ +--------------+ | |
480 | | lttng A |-------- ops ------->| ltt-sessiond | | |
481 | +-----------+ ^ +--------------+ | |
482 | | ^ commands | |
483 | +-----------+ | +-------+<--+ | |
484 | | lttng B |------+ +--->| app_1 |------- write -----+ | |
485 | +-----------+ | +-------+ | | |
486 | | | | |
487 | +-----------------+ | +-------------+ | | |
488 | | ust-consumerd A |--O--- read ----->| shared mem. |<-+ | |
489 | +-----------------+ | +-------------+ | | |
490 | | | | |
491 | +-----------------+ v +-------------+ | | |
492 | | ust-consumerd B |--+--- read ----->| shared mem. |<-+ | |
493 | +-----------------+ +-------------+ | |
494 | ||
495 | ust-consumerd A - UID: user A (rw-), GID: tracing (r--) | |
496 | ust-consumerd B - UID: user B (rw-), GID: tracing (r--) | |
497 | ||
498 | Scenario 4 - User not in the tracing group | |
499 | ------ | |
500 | ||
501 | For this particular case, it's all goes back to the first scenario. The user | |
502 | MUST start the application using his credentials. The session will be created | |
503 | by the per-user ltt-sessiond but he will not be able to trace anything that the | |
504 | user does not owned. |