From 2ddbdc7a9c2d62a7691578418d4e340c5b121972 Mon Sep 17 00:00:00 2001 From: Francis Deslauriers Date: Mon, 27 Sep 2021 09:42:54 -0400 Subject: [PATCH] Fix: notification-thread: handling event from a removed tracer event src MIME-Version: 1.0 Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit Issue ===== The issue is caused by a race condition where the `lttng_poll_wait()` returns a _REMOVE_TRACER_EVENT_SOURCE event followed by an actual notification event on the removed event source fd. This causes the notification thread to remove the fd from the potential notification sources list and later fail to find that same fd in the next iteration. This race condition can lead to the notification thread to hang indefinitely or to failed assertions within the `fini_thread_state()` function. Fix === When removing an tracer event source, force the notification thread `lttng_poll_wait()` loop to restart to ignore events from the removed fd. Use the `restart_poll` for that purpose (see note below). Reproducer ========== It's easy to reproduce this issue by adding a `usleep(5000)` just before the `lttng_poll_wait()` call in the notification thread. Note ==== It's the second time that I fix this issue. It was first fixed by this commit by adding the `restart_poll` flag: commit 8b5240601e4ddf6127e4291b7194dd5179cb35b5 Author: Francis Deslauriers Date: Thu Dec 10 15:41:29 2020 -0500 notification-thread: drain all tracer notification on removal and later, that other commit refactored that code but accidently removed the use of the `restart_poll`: commit 34bf4f69e49d8a69331a6aa6826ef1f155e20ede Author: Francis Deslauriers Date: Wed May 26 16:05:16 2021 -0400 notification-thread: remove fd from pollset on LPOLLHUP and friends Signed-off-by: Francis Deslauriers Signed-off-by: Jérémie Galarneau Change-Id: I6da0ed4374b612934adc72fb88d5c142505c5d53 --- src/bin/lttng-sessiond/notification-thread-events.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/bin/lttng-sessiond/notification-thread-events.c b/src/bin/lttng-sessiond/notification-thread-events.c index 38eb85a1e..a295739f2 100644 --- a/src/bin/lttng-sessiond/notification-thread-events.c +++ b/src/bin/lttng-sessiond/notification-thread-events.c @@ -2151,6 +2151,12 @@ int remove_tracer_event_source_from_pollset( source_element->is_fd_in_poll_set = false; + /* + * Force the notification thread to restart the poll() loop to ensure + * that any events from the removed fd are removed. + */ + state->restart_poll = true; + ret = drain_event_notifier_notification_pipe(state, source_element->fd, source_element->domain); if (ret) { -- 2.34.1