From 3b33e9e731f2091e8aa13ea035c295ed6f101eac Mon Sep 17 00:00:00 2001 From: =?utf8?q?J=C3=A9r=C3=A9mie=20Galarneau?= Date: Mon, 1 Apr 2019 18:03:57 -0400 Subject: [PATCH] Move completed trace archive chunks to an "archives" sub-folder MIME-Version: 1.0 Content-Type: text/plain; charset=utf8 Content-Transfer-Encoding: 8bit Users have expressed the desire to read all completed trace archive chunks at once. Initialy, the requirement called for users to wait, using the notification API, for chunks to be made available following the completion of a rotation. Upon the reception of a rotation completion notification, which contains the chunk's location, it would be possible to safely consume the resulting trace archive chunk. Given that using the notification API was deemed too complex for certain users, it was decided that the creation of a folder of the form "--" must guarantee the readability of its contents. This guarantee is currently not honored for the first trace archive chunk. This is a known bug which will be addressed before the release. This requirement has evolved and it must now be possible for users to point a reader to a session output directory, after an arbitrary number of rotations, to consume all completed trace archive chunks while tracing is ongoing. To make this possible (and reliable), a reader must have a way to infer which trace archive chunks are still being produced, and which have been completed so as to not attempt to consume a trace that is still being produced. First, a quick refresher on the hierachy of a session output path. Before a rotation occurs, a session output path has the following hierarchy: my_trace |__ust |__[...] |__kernel |__[...] And, after four completed rotations: my_trace |__--0 <--- completed trace archive chunk |__--1 |__--2 |__--3 |__-4 <--- trace archive being produced As a consequence of this behaviour, it is not possible to safely point a CTF reader to a trace output directory without a special configuration option that would indicate that the reader should ignore the usual lttng hierarchy ('ust' and 'kernel'). Indeed, it is not possible to distinguish a completed trace from a trace being produced before the completion of the first rotation of a session. Moreover, relying on the format of the name of trace archive chunks to infer their completeness is awfully restrictive in terms of our ability to alter the trace archive chunk directory format in the future. This format is also not part of any CTF specification document meaning that implementing this logic in a reference CTF implementation (babeltrace) is ill-advised. It would be unexpected for a reader to fail to read a trace simply because it is stored in a directory of the form -, which is a valid trace output directory. Hence, this commit changes the hierarchy of the trace output directory so that completed (and safe to read) trace archive chunks are stored in an "archives" sub-folder under the tracing session output directory. This results in the following trace output directory hierarchy: my_trace |__archives | |__--0 <--- completed trace archive chunk | |__--1 | |__--2 | |__--3 |__-4 <--- trace archive being produced Pointing to the "archives" directory requires no LTTng-specific logic to be implemented in the reader(s) to achieve the users' requirement. It does require that users wait for the presence of an "archives" directory in the trace output path. Reference shell code is provided to implement such a check: """ if [ -d ${trace_output_path}/archives ]; then # invoke reader fi """ Given that a user would have to wait for a first completed chunk to appear or ensure (in whichever way) that a rotation has occured before consuming trace archive chunks, this alternative does not seem more complex. Signed-off-by: Jérémie Galarneau --- src/bin/lttng-sessiond/rotate.c | 4 ++-- .../regression/tools/rotation/rotate_utils.sh | 5 +++++ tests/regression/tools/rotation/test_kernel | 8 ++++---- tests/regression/tools/rotation/test_ust | 18 ++++++++++-------- 4 files changed, 21 insertions(+), 14 deletions(-) diff --git a/src/bin/lttng-sessiond/rotate.c b/src/bin/lttng-sessiond/rotate.c index 7abfaed64..c3413a2ca 100644 --- a/src/bin/lttng-sessiond/rotate.c +++ b/src/bin/lttng-sessiond/rotate.c @@ -218,7 +218,7 @@ int rename_completed_chunk(struct ltt_session *session, time_t ts) * session_root_path, so we need to create the chunk folder * and move the domain-specific folders inside it. */ - ret = snprintf(new_path, sizeof(new_path), "%s/%s-%s-%" PRIu64, + ret = snprintf(new_path, sizeof(new_path), "%s/archives/%s-%s-%" PRIu64, session->rotation_chunk.current_rotate_path, start_time, datetime, session->current_archive_id); @@ -275,7 +275,7 @@ int rename_completed_chunk(struct ltt_session *session, time_t ts) ret = -1; goto end; } - ret = snprintf(new_path, sizeof(new_path), "%s/%s-%s-%" PRIu64, + ret = snprintf(new_path, sizeof(new_path), "%s/archives/%s-%s-%" PRIu64, session_get_base_path(session), start_datetime, datetime, session->current_archive_id); diff --git a/tests/regression/tools/rotation/rotate_utils.sh b/tests/regression/tools/rotation/rotate_utils.sh index 3d622deca..3cd9fab6f 100644 --- a/tests/regression/tools/rotation/rotate_utils.sh +++ b/tests/regression/tools/rotation/rotate_utils.sh @@ -116,6 +116,11 @@ function rotate_timer_test () nr_iter=0 expected_chunks=3 + # Wait for the "archives" folder to appear after the first rotation + until [ -d $local_path ]; do + sleep 1 + done + # Wait for $expected_chunks to be generated, timeout after # 3 * $expected_chunks * 0.5s. # On a laptop with an empty session, a local rotation takes about 200ms, diff --git a/tests/regression/tools/rotation/test_kernel b/tests/regression/tools/rotation/test_kernel index d7f04bed0..9a634b70b 100755 --- a/tests/regression/tools/rotation/test_kernel +++ b/tests/regression/tools/rotation/test_kernel @@ -55,7 +55,7 @@ function test_kernel_streaming () { diag "Test kernel streaming with session rotation" create_lttng_session_uri $SESSION_NAME net://localhost - rotate_kernel_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*" + rotate_kernel_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*/archives" } function test_kernel_local () @@ -63,7 +63,7 @@ function test_kernel_local () diag "Test kernel local with session rotation" create_lttng_session_ok $SESSION_NAME $TRACE_PATH - rotate_kernel_test "${TRACE_PATH}" + rotate_kernel_test "${TRACE_PATH}/archives" } function test_kernel_local_timer () @@ -74,7 +74,7 @@ function test_kernel_local_timer () lttng_enable_rotation_timer_ok $SESSION_NAME 500ms start_lttng_tracing_ok $SESSION_NAME - rotate_timer_test "${TRACE_PATH}" 0 + rotate_timer_test "${TRACE_PATH}/archives" 0 } function test_kernel_streaming_timer () @@ -85,7 +85,7 @@ function test_kernel_streaming_timer () lttng_enable_rotation_timer_ok $SESSION_NAME 500ms start_lttng_tracing_ok $SESSION_NAME - rotate_timer_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*" 0 + rotate_timer_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*/archives" 0 } plan_tests $NUM_TESTS diff --git a/tests/regression/tools/rotation/test_ust b/tests/regression/tools/rotation/test_ust index 69bc18e1c..a4988dada 100755 --- a/tests/regression/tools/rotation/test_ust +++ b/tests/regression/tools/rotation/test_ust @@ -75,7 +75,7 @@ function test_ust_streaming_uid () create_lttng_session_uri $SESSION_NAME net://localhost enable_ust_lttng_event_ok $SESSION_NAME $EVENT_NAME - rotate_ust_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*" "ust/uid/*/*/" 0 + rotate_ust_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*/archives" "ust/uid/*/*/" 0 } function test_ust_local_uid () @@ -84,7 +84,7 @@ function test_ust_local_uid () create_lttng_session_ok $SESSION_NAME $TRACE_PATH enable_ust_lttng_event_ok $SESSION_NAME $EVENT_NAME - rotate_ust_test "${TRACE_PATH}" "ust/uid/*/*/" 0 + rotate_ust_test "${TRACE_PATH}/archives" "ust/uid/*/*/" 0 } function test_ust_streaming_pid () @@ -94,7 +94,7 @@ function test_ust_streaming_pid () enable_channel_per_pid $SESSION_NAME "channel0" enable_ust_lttng_event_ok $SESSION_NAME $EVENT_NAME "channel0" - rotate_ust_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*" "ust/pid/*/" 1 + rotate_ust_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*/archives" "ust/pid/*/" 1 } function test_ust_local_pid () @@ -104,7 +104,7 @@ function test_ust_local_pid () enable_channel_per_pid $SESSION_NAME "channel0" enable_ust_lttng_event_ok $SESSION_NAME $EVENT_NAME "channel0" - rotate_ust_test "${TRACE_PATH}" "ust/pid/*/" 1 + rotate_ust_test "${TRACE_PATH}/archives" "ust/pid/*/" 1 } function test_ust_local_timer_uid () @@ -117,7 +117,7 @@ function test_ust_local_timer_uid () # We just want the app to register, no event generated $TESTAPP_BIN 0 0 /dev/null 2>&1 - rotate_timer_test "${TRACE_PATH}" 0 + rotate_timer_test "${TRACE_PATH}/archives" 0 } function test_ust_streaming_timer_uid () @@ -130,7 +130,7 @@ function test_ust_streaming_timer_uid () # We just want the app to register, no event generated $TESTAPP_BIN 0 0 /dev/null 2>&1 - rotate_timer_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*" 0 + rotate_timer_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*/archives" 0 } function test_ust_local_timer_pid () @@ -144,7 +144,7 @@ function test_ust_local_timer_pid () # We just want the app to register, no event generated $TESTAPP_BIN 0 0 /dev/null 2>&1 - rotate_timer_test "${TRACE_PATH}" 1 + rotate_timer_test "${TRACE_PATH}/archives" 1 } function test_ust_streaming_timer_pid () @@ -158,7 +158,7 @@ function test_ust_streaming_timer_pid () # We just want the app to register, no event generated $TESTAPP_BIN 0 0 /dev/null 2>&1 - rotate_timer_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*" 1 + rotate_timer_test "${TRACE_PATH}/${HOSTNAME}/${SESSION_NAME}*/archives" 1 } function test_incompatible_sessions () @@ -207,6 +207,8 @@ tests=( test_ust_streaming_uid test_ust_local_uid \ test_ust_local_timer_pid test_ust_streaming_timer_pid \ test_incompatible_sessions ) +#tests=( test_ust_local_timer_uid ) + for fct_test in ${tests[@]}; do SESSION_NAME=$(randstring 16 0) -- 2.34.1