--- /dev/null
+
+Some thoughts about userspace tracing
+
+Mathieu Desnoyers January 2006
+
+
+
+* Goals
+
+Fast and secure user space tracing.
+
+Fast :
+
+- 5000ns for a system call is too long. Writing an event directly to memory
+ takes 220ns.
+- Still, we can afford a system call for buffer switch, which occurs less often.
+- No locking, no signal disabling. Disabling signals require 2 system calls.
+ Mutexes are implemented with a short spin lock, followed by a yield. Yet
+ another system call. In addition, we have no way to know on which CPU we are
+ running when in user mode. We can be preempted anywhere.
+- No contention.
+- No interrupt disabling : it doesn't exist in user mode.
+
+Secure :
+
+- A process shouldn't be able to corrupt the system's trace or another
+ process'trace. It should be limited to its own memory space.
+
+
+
+* Solution
+
+- Signal handler concurrency
+
+Using atomic space reservation in the buffer(s) will remove the requirement for
+locking. This is the fast and safe way to deal with concurrency coming from
+signal handlers.
+
+- Start/stop tracing
+
+Two possible solutions :
+
+Either we export a read-only memory page from kernel to user space. That would
+be somehow seen as a hack, as I have never even seen such interface anywhere
+else. It may lead to problems related to exported types. The proper, but slow,
+way to do it would be to have a system call that would return the tracing
+status.
+
+My suggestion is to go for a system call, but only call it :
+
+- when the process starts
+- when receiving a SIG_UPDTRACING
+
+Two possibilities :
+
+- one system call per information to get/one system call to get all information.
+- one signal per information to get/one signal for "update" tracing info.
+
+I would tend to adopt :
+
+- One signal for "general tracing update"
+ One signal handler would clearly be enough, more would be unnecessary
+ overhead/pollution.
+- One system call for all updates.
+ We will need to have multiple parameters though. We have up to 6 parameters.
+
+syscall get_tracing_info
+
+first parameter : active traces mask (32 bits : 32 traces).
+
+
+Concurrency
+
+We must have per thread buffers. Then, no memory can be written by two threads
+at once. It removes the need for locks (ok, atomic reservation was already doing
+that) and removes false sharing.
+
+
+Multiple traces
+
+By having the number of active traces, we can allocate as much buffers as we
+need. The only thing is that the buffers will only be allocated when receiving
+the signal/starting the process and getting the number of traces actives.
+
+It means that we must make sure to only update the data structures used by
+tracing functions once the buffers are created.
+
+When adding a new buffer, we should call the set_tracing_info syscall and give
+the new buffers array to the kernel. It's an array of 32 pointers to user pages.
+They will be used by the kernel to get the last pages when the thread dies.
+
+If we remove a trace, the kernel should stop the tracing, and then get the last
+buffer for this trace. What is important is to make sure no writers are still
+trying to write in a memory region that get desallocated.
+
+For that, we will keep an atomic variable "tracing_level", which tells how many
+times we are nested in tracing code (program code/signal handlers) for a
+specific trace.
+
+We could do that trace removal in two operations :
+
+- Send an update tracing signal to the process
+ - the sig handler get the new tracing status, which tells that tracing is
+ disabled for the specific trace. It writes this status in the tracing
+ control structure of the process.
+ - If tracing_level is 0, well, it's fine : there are no potential writers in
+ the removed trace. It's up to us to buffer switch the removed trace, and,
+ after the control returns to us, set_tracing_info this page to NULL and
+ delete this memory area.
+ - Else (tracing_level > 0), flag the removed trace for later switch/delete.
+
+ It then returns control to the process.
+
+- If the tracing_level was > 0, there was one or more writers potentially
+ accessing this memory area. When the control comes back to the writer, at the
+ end of the write in a trace, if the trace is marked for switch/delete and the
+ tracing_level is 0 (after the decrement of the writer itself), then the
+ writer must buffer switch, set_tracing_info to NULL and then delete the
+ memory area.
+
+
+Filter
+
+The update tracing info signal will make the thread get the new filter
+information. Getting this information will also happen upon process creation.
+
+parameter 2 for the get tracing info : array of 32 ints (32 bits).
+Each integer is the filter mask for a trace. As there are up to 32 active
+traces, we have 32 integers for filter.
+
+
+Buffer switch
+
+There could be a tracing_buffer_switch system call, that would give the page
+start address as parameter. The job of the kernel is to steal this page,
+possibly replacing it with a zeroed page (we don't care about the content of the
+page after the syscall).
+
+Process dying
+
+The kernel should be aware of the current pages used for tracing in each thread.
+If a thread dies unexpectedly, we want the kernel to get the last bits of
+information before the thread crashes.
+
+syscall set_tracing_info
+
+parameter 1 : array of 32 user space pointers to current pages or NULL.
+
+
+Memory protection
+
+We want each process to be usable to make a trace unreadable, and each process
+to have its own memory space.
+
+Two possibilities :
+
+Either we create one channel per process, or we have per cpu tracefiles for all
+the processes, with the specification that data is written in a monotically
+increasing time order and that no process share a 4k page with another process.
+
+The problem with having only one tracefile per cpu is that we cannot safely
+steal a process'buffer upon a schedule change because it may be currently
+writing to it.
+
+It leaves the one tracefile per thread as the only solution.
+
+Another argument in favor of this solution is the possibility to have mixed
+32-64 bits processes on the same machine. Dealing with types will be easier.
+
+
+Corrupted trace
+
+A corrupted tracefile will only affect one thread. The rest of the trace will
+still be readable.
+
+
+Facilities
+
+Upon process creation or when receiving the signal of trace info update, when a
+new trace appears, the thread should write the facility information into it. It
+must then have a list of registered facilities, all done at the thread level.
+
+We must decide if we allow a facility channel for each thread. The advantage is
+that we have a readable channel in flight recorder mode, while the disadvantage
+is to duplicate the number of channels, which may become quite high. To follow
+the general design of a high throughput channel and a low throughput channel for
+vital information, I suggest to have a separate channel for facilities, per
+trace, per process.
+
+
+
+
+
+
projects / lttv.git / commitdiff
