doc/session-daemon-model.txt

   1 RFC - New processes model for UST and LTTng
   2
   3 Author: David Goulet <david.goulet@polymtl.ca>
   4
   5 Contributors:
   6     * Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
   7     * Yannick Brosseau <yannick.brosseau@polymtl.ca>
   8     * Nils Carlson <nils.carlson@ericsson.com>
   9     * Michel Dagenais <michel.dagenais@polymtl.ca>
  10     * Stefan Hajnoczi <stefanha@gmail.com>
  11
  12 Version:
  13     - v0.1: 17/01/2011
  14         * Initial proposal
  15
  16     - v0.2: 19/01/2011
  17         After multiple reply from all the contributors above, here is the list
  18         of what has changed:
  19         * Change/Add Terminology elements from the initial model
  20         * New figures for four new scenarios
  21         * Add inprocess library section
  22         * LTTng kernel tracer support proposition
  23         * More details for the Model and Components
  24         * Improve the basic model. Quite different from the last one
  25
  26     - v0.3: 28/01/2011
  27         In response from Michel Dagenais and Nils Carlson comments:
  28         * Add scaling to reasons of this re-engineering
  29         * Purpose of the session ID
  30         * Explain why ltt-sessiond creates the tracing buffers
  31         * ust-consumerd interaction schema
  32         * Clarify inprocess library behavior
  33
  34     - v0.4: 01/02/2011
  35         After Mathieu Desnoyers and Michel Dagenais comments:
  36         * Add section Introduction
  37         * Define the global and per-user ltt-sessiond
  38         * Add details for ltt-sessiond in the inprocess lib section
  39         * Session ID are now UUID
  40         * Add buffer snapshot schema for ust-consumerd
  41         * ltt-sessiond validate inprocess lib version
  42         * ltt-sessiond socket validation by the inprocess lib.
  43         * Add lttng definition
  44         * Add consumer to the Model section
  45
  46 Terminology
  47 -----------------
  48
  49 ltt-sessiond - Main daemon for trace session registry for UST and LTTng
  50 NOTE: Changed to lttng-sessiond in the git tree
  51
  52 ust-consumerd - Daemon that consume UST buffers for a specific application
  53
  54 ltt-consumerd - Daemon that consume LTTng buffers
  55
  56 tracing session - A trace linked to a set of specific tracepoints and to a set
  57                   of tracing buffers
  58
  59 tracing buffers - Buffers containing tracing data
  60
  61 tracing data - Data created by tracing an application
  62
  63 inprocess library - UST library linked with the application
  64
  65 shared memory - system V shared memory
  66
  67 application common named pipe - Global named pipe that triggers application
  68                                 registration, on pipe event, to ltt-sessiond
  69
  70 lttng - New command line tool for LTTng and UST tracing control
  71
  72 Introduction
  73 -----------------
  74
  75 This RFC propose a brand new UST and LTTng daemon model. This re-engineering
  76 was mostly driven by the need of:
  77
  78     * Better security in terms of access rights on tracing data
  79     * Manage tracing session
  80     * Scaling in terms of thread/processes needed to perform tracing
  81     * LTTng and UST integration in terms of merging traces and session control
  82     * Networking such as streaming and remote control over different traces
  83
  84 The new model follows the basic principles of having a session registry
  85 (ltt-sessiond) and consumers for each tracing session (ust-consumerd and
  86 ltt-consumerd).
  87
  88 With this proposal, LTTng and UST will share the same tracing session, be
  89 managed by the same tool and bring a complete integration between these two
  90 powerful tools.
  91
  92 NOTE: This proposal does NOT makes UST dependent on LTTng and vice versa.
  93
  94 Model
  95 -----------------
  96
  97 A global and/or per-user registry keeps track of all tracing sessions. Any user
  98 that wants to manage either a kernel trace using LTTng or an application trace
  99 with UST must interact with that registry for any possible actions.
 100
 101 The model address multiple tracing use cases based on the fact that we
 102 introduce a tracing Unix group (tracing group). Only users in that group or
 103 root can use the global registry. Other users will create a local registry
 104 (per-user registry) that will be completely independent from the global one.
 105
 106 Two cases:
 107
 108     1) Users in the tracing group, it's tracing session can consume all tracing
 109        buffers from all applications and the kernel.
 110
 111     2) Users NOT in the tracing group, it's tracing session can only consume
 112        data from its own applications' buffers hence tracing his applications.
 113
 114 A session stored by the registry consist of:
 115
 116     * Session name (given by the user or automatically assigned)
 117     * List of traces (LTTng or UST)
 118     * Tracepoints/markers associated to a trace of that session
 119     * UUID
 120     * Associated user (UID)
 121
 122 Then, consumers are used to extract data from tracing buffers. These consumers
 123 are daemon consuming either UST or/and LTTng buffers. For a single session,
 124 only one UST consumer and one LTTng consumer is necessary. The daemon CAN
 125 handle multiple tracing buffers for network streaming by example or for quick
 126 snapshot. These consumers are told by the inprocess library or the kernel to
 127 start getting out data on disk or network.
 128
 129 For the next subsections, every components of this new proposal is explained
 130 from the global and per-user registry perspective.
 131
 132 LTT-SESSIOND:
 133
 134 The ltt-sessiond daemon acts as a session registry i.e. by keeping reference to
 135 all active session and, by active, it means a session in any state other than
 136 destroyed. Each entity we are keeping track of, here session, will have a
 137 universal unique identifier (UUID) assigned to it. The purpose of this UUID is
 138 to track a session in order to apply any kind of actions (Ex: Attach, Destroy).
 139 A human readable version SHOULD be consider in order to facilitate the session
 140 identification when listed by lttng.
 141
 142 The daemon creates two local Unix sockets (AF_UNIX). The first one is for what
 143 we call client communication i.e. interaction with lttng (or any other
 144 compatible tools). That socket is set with the ltt-sessiond credentials with
 145 read-write mode for both user and group. The second one is a global socket for
 146 application registration for the UST case (see inprocess lib subsection below).
 147
 148 This daemon is also responsible for tracing buffers creation. Two main reasons
 149 motivate this design:
 150
 151     * The ltt-sessiond needs to keep track of all the shared memory segments in
 152     order to be able to give reference to any other possible consumer.
 153
 154     * For the case of sharing tracing buffers between all userspace
 155     applications, having the registry allocating them will allow that but, if
 156     the inprocess library was allocating them, we will need to redesign the
 157     whole model.
 158
 159 For all tracing actions either to interact with a session or a specific trace,
 160 the lttng client MUST go through ltt-sessiond. The daemon will take care of
 161 routing the command to the write inprocess library or the kernel.
 162
 163 Global registry:
 164
 165 A global registry SHOULD be started, idealy at boot, with credentials UID root
 166 and GID of the tracing group. Only user within the tracing group will be able
 167 to interact with that registry. All applications will try to register to that
 168 registry using the global socket (second one discuss above).
 169
 170 Per-user registry:
 171
 172 This type of registry address two use cases. The first one is when a session
 173 creation is requested from lttng but no global ltt-sessiond exist. So, a
 174 ltt-sessiond will be spawned in order to manage the tracing of that user. The
 175 second use case is when a user is not in the tracing group thus he cannot
 176 communication with the global registry.
 177
 178 However, care MUST be put in order to manage the socket's daemon. They are not
 179 global anymore so they should be created in the home directory of the user
 180 requesting tracing.
 181
 182 In both cases, for global and per-user registry, all applications MUST try to
 183 register to both ltt-sessiond. (see inprocess library subsection for details)
 184
 185 The trace roles of ltt-sessiond:
 186
 187     Trace interaction - Create, Destroy, Pause, Stop, Start, Set options
 188
 189     Registry - keep track of trace's information:
 190         * shared memory location (only the keyid)
 191         * application PID (UST)
 192         * type (kernel or UST)
 193         * session name
 194         * UID
 195
 196     Buffers creation - creates shared memory for the tracing buffers.
 197
 198 UST-CONSUMERD:
 199
 200 The purpose of this daemon is to consume the UST trace buffers for only a
 201 specific session. The session MAY have several traces for example two different
 202 applications. The client tool, lttng has to create the ust-consumerd if NONE
 203 is available for that session. It is very important to understand that for a
 204 tracing session, there is only one ust-consumerd for all the traced
 205 applications.
 206
 207 This daemon basically empty the tracing buffers when asked for and writes that
 208 data to disk for future analysis using LTTv or/and TMF (Tracing Monitoring
 209 Frameworks). The inprocess library is the one that tells the ust-consumerd
 210 daemon that the buffers are ready for consumption.
 211
 212 Here is a flow of action to illustrate the ust-consumerd life span:
 213
 214 1)
 215 +-----------+    ops     +--------------+
 216 | lttng A |<---------->| ltt-sessiond |
 217 +-----------+            +--------------+
 218
 219 lttng ask for tracing an application using the PID and the session UUID. The
 220 shared memory reference is given to lttng and the ust-consumerd communication
 221 socket if ust-consumerd already exist.
 222
 223 2a) If ust-consumerd EXIST
 224
 225 +-----------+
 226 | lttng A |
 227 +-----------+
 228     | mem ref.
 229     |   +---------------+   read   +------------+
 230     +-->| ust-consumerd |--------->| shared mem |
 231         +---------------+          +------------+
 232
 233 In that case, lttng only ask ust-consumerd to consume the buffers using
 234 the reference it previously got from ltt-sessiond.
 235
 236 2b) If ust-consumerd DOES NOT EXIST
 237
 238 +-----------+              +--------------+
 239 | lttng A |        +---->| ltt-sessiond |
 240 +-----------+        |     +--------------+
 241     | ID             |
 242     | mem ref.       | register
 243     |   +---------------+
 244     +-->| ust-consumerd |
 245         +---------------+
 246
 247 lttng spawns the ust-consumerd for the session using the session UUID in
 248 order for the daemon to register as a consumer to ltt-sessiond for that
 249 session.
 250
 251 Quick buffer snapshot:
 252
 253 1) Here, lttng will request a buffer snapshot for an already running session.
 254
 255 +-----------+                     +--------------+
 256 | lttng A |-------- ops ------->| ltt-sessiond |
 257 +-----------+                     +--------------+
 258  |                                      | command
 259  |  +-----------------+     +-------+<--+
 260  |  | ust-consumerd 1 |<----| app_1 |-+
 261  |  +-----------------+     +-------+ | write
 262  |    1   |                           v
 263  |        |               +-------------+
 264  |        +--- read ----->| shared mem. |
 265  |                        +-------------+
 266  |                               ^
 267  |  +-----------------+          |
 268  +->| ust-consumerd 2 |----------+
 269     +-----------------+  snapshot
 270             | write
 271             |
 272             +---> disk/network
 273
 274 The first ust-consumerd (1) was already consuming buffers for the current
 275 session. So, lttng ask for a live snapshot. A new ust-consumerd (2) is
 276 spawned, snapshot the buffers using the shared memory reference from
 277 ltt-sessiond, writes date to disk and die after all.
 278
 279 On the security side, the ust-consumerd gets UID/GID from the lttng
 280 credentials since it was spawned by lttng and so the files containing the
 281 tracing data will also be set to UID/GID of the lttng client. No setuid or
 282 setgid is used, we only use the credentials of the user.
 283
 284 The roles of ust-consumerd:
 285
 286     Register to ltt-sessiond - Using a session UUID and credentials (UID/GID)
 287
 288     Consume buffers - Write data to a file descriptor (on disk, network, ...)
 289
 290 Buffer consumption is triggered by the inprocess library which tells
 291 ust-consumerd when to consume.
 292
 293 LTT-CONSUMERD:
 294
 295 The purpose of this daemon is to consume the LTTng trace buffers for only a
 296 specific session.
 297
 298 For that kernel consumer, ltt-sessiond will pass different anonymous file
 299 descriptors to the ltt-consumerd using a Unix socket. From these file
 300 desriptors, it will be able to get the data from a special function export by
 301 the LTTng kernel.
 302
 303 ltt-consumerd will be managed by the exact same way as ust-consumerd. However,
 304 in order to trace the kernel, you are either root (UID=0) or in the tracing
 305 group.
 306
 307 The roles of ltt-consumerd:
 308
 309     Register to ltt-sessiond - Using a session UUID and credentials (UID/GID)
 310
 311     Consume buffers - Write data to a file descriptor (on disk, network, ...)
 312
 313 Kernel triggers ltt-consumerd for buffer consumption.
 314
 315 UST INPROCESS LIBRARY:
 316
 317 When the application starts, this library will check for the global named pipe
 318 of ltt-sessiond. If present, it MUST validate that root is the owner. This
 319 check is very important to prevent ltt-sessiond spoofing. If the pipe is root,
 320 we are certain that it's the privileged user that operates tracing. Then, using
 321 it's UID, the application will try to register to the per-user ltt-sessiond
 322 again verifying before the owner ship of the named pipe that should match the
 323 UID.
 324
 325 Before registration, the inprocess library MUST validate with the ltt-sessiond
 326 the library version for compatibility reason. This is mechanism is useful for
 327 library compatibility but also to see if ltt-sessiond socket is valid (means
 328 that an actual ltt-sessiond is listening on the other side). Having no response
 329 for over 10 seconds, the application will cut communication on that socket and
 330 fallback to the application common named pipe (explain below).
 331
 332 If the socket is valid, it will register as a traceable application using the
 333 apps credentials and will open a local Unix socket, passed to ltt-sessiond, in
 334 order to receive an eventual shared memory reference. It will then wait on it
 335 if any other command are given by the lttng client. This socket becomes the
 336 only channel of communication between the registry and the application.
 337
 338 If no ltt-sessiond is present at registration, the application tries to open
 339 the application common named pipe or create it if it does not exist and wait on
 340 it (using poll or epoll Linux API). Having any type of event on that pipe, the
 341 inprocess library will then try to register to the global and per-user
 342 ltt-sessiond. If it fails again, it goes back again to wait on that pipe.
 343
 344 SHARED MEMORY
 345
 346 For UST, this is the memory area where the tracing buffers will be held and
 347 given access in read-write mode for the inprocess library of the application.
 348
 349 On the LTTng side (for ltt-consumerd), these buffers are in the kernel space
 350 and given access by opening a file in the debugfs file system. With an
 351 anonymous file desriptor, this consumer will be able to extract the data.
 352
 353 This memory is ONLY used for the tracing data. No communication between
 354 components is done using that memory.
 355
 356 A shared memory segment for tracing MUST be set with the tracing group GID for
 357 the UST buffers. This is the job of ltt-sessiond.
 358
 359 PREREQUISITES:
 360
 361 The global ltt-sessiond daemon MUST always be running as "root" or an
 362 equivalent user having the same privilege as root (UID = 0).
 363
 364 The ltt-sessiond daemon SHOULD be up and running at all time in order to trace
 365 a tracable application.
 366
 367 The new lttng library API MUST be used to interact with the
 368 ltt-sessiond registry daemon for every trace action needed by the user.
 369
 370 A tracing group MUST be created. Whoever is in that group is able to access the
 371 tracing data of any buffers and is able to trace any application or the kernel.
 372
 373 WARNING: The tracing group name COULD interfere with other already existing
 374 groups. Care should be put at install time for that (from source and packages)
 375
 376 The next section illustrates different use cases using that new model.
 377
 378 Use Cases
 379 -----------------
 380
 381 Each case considers these :
 382
 383 * user A - UID: A; GID: A, tracing
 384 * user B - UID: B; GID: B, tracing
 385
 386 Scenario 1 - Single user tracing app_1
 387 ------
 388
 389 This first scenario shows how user A will start a trace for application app_1
 390 that is not running.
 391
 392 1) lttng ask ltt-sessiond for a new session through a Unix socket. If
 393 allowed, ltt-sessiond returns a session UUID to the client.
 394 (Ex: ops --> new session)
 395
 396 +-----------+    ops     +--------------+
 397 | lttng A   |<---------->| ltt-sessiond |
 398 +-----------+            +--------------+
 399
 400 2) The app_1 is spawned by lttng having the user A credentials. Then, app_1
 401 automatically register to ltt-sessiond has a "tracable apps" through the global
 402 named pipe of ltt-sessiond using the UID/GID and session UUID.
 403
 404 The shared memory is created with the app_1 UID (rw-) and tracing group GID
 405 (r--) and a reference is given back to app_1
 406
 407 +-----------+            +--------------+
 408 | lttng A   |            | ltt-sessiond |
 409 +-----------+            +--------------+
 410     |                     ^    |
 411     |   +-------+         |    |   +-------------+
 412     +-->| app_1 |<--------+    +-->| shared mem. |
 413         +-------+                  +-------------+
 414
 415 3) app_1 connect to the shared memory and ust-consumerd is spawned with the
 416 session UUID and lttng credentials (user A). It then register to ltt-sessiond
 417 for a valid session to consume using the previous session UUID and credentials.
 418
 419 +-----------+            +--------------+
 420 | lttng A   |        +-->| ltt-sessiond |----------+
 421 +-----------+        |   +--------------+          |
 422     |                |                             |
 423     |   +---------------+    read                  | commands
 424     +-->| ust-consumerd |---------+                |   and
 425         +---------------+         v                | options
 426           ^  |                  +-------------+    |
 427           |  v          +------>| shared mem. |    |
 428         +-------+       |       +-------------+    |
 429         | app_1 |--------                          |
 430         +-------+     write                        |
 431             ^                                      |
 432             +---------------------------------------
 433
 434 Scenario 2 - Single user tracing already running app_1
 435 ------
 436
 437 1) lttng ask ltt-sessiond for a new session through a Unix socket. If allowed
 438 (able to write on socket), ltt-sessiond returns a session UUID to the client.
 439
 440 +-----------+    ops     +--------------+
 441 | lttng A   |<---------->| ltt-sessiond |
 442 +-----------+            +--------------+
 443                            ^
 444         +-------+   read   |
 445         | app_1 |----------+
 446         +-------+
 447
 448 NOTE: At this stage, since app_1 is already running, the registration of app_1
 449 to ltt-sessiond has already been done. However, the shared memory segment is
 450 not allocated yet until a trace session is initiated. Having no shared memory,
 451 the inprocess library of app_1 will wait on the local Unix socket connected to
 452 ltt-sessiond for the reference.
 453
 454 +-----------+            +--------------+
 455 | lttng A   |            | ltt-sessiond |
 456 +-----------+            +--------------+
 457                           ^    |
 458         +-------+         |    |   +-------------+
 459         | app_1 |<--------+    +-->| shared mem. |
 460         +-------+    commands      +-------------+
 461             |                           ^
 462             +---------- write ----------+
 463
 464 2) lttng spawns a ust-consumerd for the session. We get the same figure as
 465 step 3 in the first scenario.
 466
 467 There is a small difference though. The application MAY NOT be using the same
 468 credentials as user A (lttng).  However, the shared memory is always GID of
 469 the tracing group. So, in order for user A to trace app_1, is MUST be in the
 470 tracing group otherwise, if the application is not set with the user
 471 credentials, user A will not be able to trace app_1
 472
 473 Scenario 3 - Multiple users tracing the same running application
 474 ------
 475
 476 1) Session are created for the two users. Using the same exact mechanism as
 477 before, the shared memory and consumers are created. Two users, two sessions,
 478 two consumers and two shared memories for the same application.
 479
 480 +-----------+                     +--------------+
 481 | lttng A   |-------- ops ------->| ltt-sessiond |
 482 +-----------+      ^              +--------------+
 483                    |                           ^ commands
 484 +-----------+      |               +-------+<--+
 485 | lttng B   |------+          +--->| app_1 |------- write -----+
 486 +-----------+                 |    +-------+                   |
 487                               |                                |
 488          +-----------------+  |               +-------------+  |
 489          | ust-consumerd A |--O--- read ----->| shared mem. |<-+
 490          +-----------------+  |               +-------------+  |
 491                               |                                |
 492          +-----------------+  v               +-------------+  |
 493          | ust-consumerd B |--+--- read ----->| shared mem. |<-+
 494          +-----------------+                  +-------------+
 495
 496 ust-consumerd A - UID: user A (rw-), GID: tracing (r--)
 497 ust-consumerd B - UID: user B (rw-), GID: tracing (r--)
 498
 499 Scenario 4 - User not in the tracing group
 500 ------
 501
 502 For this particular case, it's all goes back to the first scenario. The user
 503 MUST start the application using his credentials. The session will be created
 504 by the per-user ltt-sessiond but he will not be able to trace anything that the
 505 user does not owned.