| 1 | LTTng Relay Daemon Architecture |
| 2 | Mathieu Desnoyers, August 2015 |
| 3 | |
| 4 | This document describes the object model and architecture of the relay |
| 5 | daemon, after the refactoring done within the commit "Fix: Relay daemon |
| 6 | ownership and reference counting". |
| 7 | |
| 8 | We have the following object composition hierarchy: |
| 9 | |
| 10 | relay connection (main.cpp, for sessiond/consumer) |
| 11 | | |
| 12 | \-> 0 or 1 per-consumer relay session |
| 13 | | |
| 14 | \-> 0 or many ctf-trace |
| 15 | | |
| 16 | \-> 0 or many stream |
| 17 | | | |
| 18 | | \-> 0 or many index |
| 19 | | |
| 20 | \-------> 0 or 1 viewer stream |
| 21 | |
| 22 | live connection (live.cpp, for client) |
| 23 | | |
| 24 | \-> 1 viewer session |
| 25 | | | |
| 26 | | \-> (transient ref) 0 or many unannounced relay streams |
| 27 | | |
| 28 | \-> 0 or many per-consumer relay sessions (actually a reference to a |
| 29 | | per-consumer relay session as created by the relay connection) |
| 30 | | |
| 31 | \-> ..... (ctf-trace, stream, index, viewer stream) |
| 32 | |
| 33 | There are global tables declared in lttng-relayd.h for per-consumer relay sessions |
| 34 | (sessions_ht, indexed by session id), streams (relay_streams_ht, indexed |
| 35 | by stream handle), viewer sessions (viewer_sessions_ht, indexed by |
| 36 | connection sock fd), and viewer streams (viewer_streams_ht, indexed |
| 37 | by stream handle). The purpose of those tables is to allow fast lookup of |
| 38 | those objects using the IDs received in the communication protocols. |
| 39 | |
| 40 | There is also one connection hash table per worker thread. There is one |
| 41 | worker thread to receive data (main.c), and one worker thread to |
| 42 | interact with viewer clients (live.c). Those tables are indexed by |
| 43 | socket file descriptor. |
| 44 | |
| 45 | A RCU lookup+refcounting scheme has been introduced for all objects |
| 46 | (except viewer session which is still an exception at the moment). This |
| 47 | scheme allows looking up the objects or doing a traversal on the RCU |
| 48 | linked list or hash table in combination with a getter on the object. |
| 49 | This getter validates that there is still at least one reference to the |
| 50 | object, else the lookup acts just as if the object does not exist. |
| 51 | |
| 52 | The relay_connection (connection between the sessiond/consumer and the |
| 53 | relayd) is the outermost object of its hierarchy. |
| 54 | |
| 55 | The live connection (connection between a live client and the relayd) |
| 56 | is the outermost object of its hierarchy. |
| 57 | |
| 58 | There is also a "lock" mutex in each object. Those are used to |
| 59 | synchronize between threads (currently the main.c relay thread and |
| 60 | live.c client thread) when objects are shared. Locks can be nested from |
| 61 | the outermost object to the innermost object. IOW, the ctf-trace lock can |
| 62 | nest within the per-consumer relay session lock. The unannounced stream list |
| 63 | lock in viewer sessions is an exception to the default locking order: it may |
| 64 | be nested inside the following locks (in order): relay session, ctf_trace, |
| 65 | and relay stream. |
| 66 | |
| 67 | RCU linked lists are used to iterate using RCU, and are protected by |
| 68 | their own mutex for modifications. Iterations should be confirmed using |
| 69 | the object "getter" to ensure its refcount is not 0 (except in cases |
| 70 | where the caller actually owns the objects and therefore can assume its |
| 71 | refcount is not 0). |
| 72 | |
| 73 | RCU hash tables are used to iterate using RCU. Iteration should be |
| 74 | confirmed using the object "getter" to ensure its refcount is not 0 |
| 75 | (except again if we have ownership and can assume the object refcount is |
| 76 | not 0). |
| 77 | |
| 78 | Object creation has a refcount of 1. Each getter increments the |
| 79 | refcount, and needs to be paired with a "put" to decrement it. A final |
| 80 | put on "self" (ownership) will allow refcount to reach 0, therefore |
| 81 | triggering release, and thus free through call_rcu. |
| 82 | |
| 83 | In the composition scheme, we find back references from each composite |
| 84 | to its container. Therefore, each composite holds a reference (refcount) |
| 85 | on its container. This allows following pointers from e.g. viewer stream |
| 86 | to stream to ctf-trace to per-consumer relay session without performing |
| 87 | any validation, due to transitive refcounting of those back-references. |
| 88 | |
| 89 | In addition to those back references, there are a few key ownership |
| 90 | references held. The connection in the relay worker thread (main.c) |
| 91 | holds ownership on the per-consumer relay session, and on each stream it |
| 92 | contains. The connection in the live worker thread (live.c) holds ownership on |
| 93 | each viewer stream it creates. The rest is ensured by back references from |
| 94 | composite to container objects. When a connection is closed, it puts all |
| 95 | the ownership references it is holding. This will then eventually |
| 96 | trigger destruction of the per-consumer relay session, streams, and |
| 97 | viewer streams associated with the connection when all the back references |
| 98 | reach 0. |
| 99 | |
| 100 | RCU read-side locks are now only held during iteration on RCU lists and |
| 101 | hash tables, and within the internals of the get (lookup) and put |
| 102 | functions. Those functions then use refcounting to ensure existence of |
| 103 | the object when returned to their caller. |
| 104 | |
| 105 | In the current implementation, while a live viewer may attach to |
| 106 | multiple per-consumer relay sessions, they are meant to belong to |
| 107 | only a single lttng-sessiond session. |