Feature request

Feature description

This is a follow-up to #957 and relates to ros2/ros2_tracing#23

I would like to discuss declaring a dependency on the LTTng userspace tracer (LTTng-UST) on Linux. I'd like to make sure that this change would be welcomed. My goal would be for this to be implemented a fair amount of time before the next release (Humble).

Core ROS 2 packages are instrumented with calls to functions provided by the tracetools package. These functions contain:

1. LTTng tracepoints if LTTng is installed/detected and tracetools is re-compiled/overlayed
2. nothing when building from source if LTTng isn't installed/detected
3. nothing in debians/Linux binaries
4. nothing in any code built for/on non-Linux platforms (macOS, Windows)

and

5. the function calls themselves can be completely removed by building instrumented core packages with -DTRACETOOLS_DISABLED=ON

This LTTng dependency isn't declared. This was initially for simplicity (especially since this is a platform-specific dependency) and because of some concerns about overhead.

This means that this feature currently isn't available out-of-the-box; users need to manually install LTTng and build from source (or at least just tracetools), since it's not installed when installing debians or running rosdep install. This has led to some questions and issues:

• https://gitlab.com/ros-tracing/ros2_tracing/-/issues/94
• https://gitlab.com/ros-tracing/ros2_tracing/-/merge_requests/139

As mentioned in #957 (comment), the LTTng userspace tracer is less "invasive" than the kernel tracer. I don't think we need to declare a dependency on the kernel tracer. Users could install it manually if they want to record kernel data for their analyses.

Also, in practice, the overhead is really negligible if tracepoints are not explicitly enabled: https://github.com/lttng/lttng-ust/blob/1f8a8ec9581af89d98aec47de9ad9e25087cd54a/include/lttng/tracepoint.h#L53-L62. Users/companies that do not want to ship a ROS 2-based product with LTTng can completely disable/remove it by setting the right CMake option (see item 5 above) and building from source, which I assume is something they are already doing.

Some references:

• Comparing overhead of tracepoints compiled in and enabled vs tracepoints compiled out for a pub->sub use-case
  • Section VI: https://arxiv.org/pdf/2201.00393.pdf
• Average overhead of a single LTTng-UST tracepoint: 158 ns (on a "typical" system)
  • Table VI: https://publications.polymtl.ca/3816/1/2018_Gebai_Survey_analysis_kernel_userspace_tracers.pdf
• Impact of a tracepoint that is compiled in but not enabled (suggested default behaviour): a simple condition check for a branch which is optimized for the case where the tracepoint is disabled
  • Section IV-B: https://publications.polymtl.ca/3816/1/2018_Gebai_Survey_analysis_kernel_userspace_tracers.pdf

Implementation considerations

New dependencies for tracetools:

• liblttng-ust-dev: LTTng-UST
• lttng-tools: LTTng CLI (not strictly necessary, but recommended to be able to actually use the tracepoints)

These would of course need to be installed on the Linux ci.ros2.org executors.

If the tracing tests turn out to create problems on ci.ros2.org, skipping them could be an option, but I think it would probably be fine.

• Also, the ros2_tracing README currently recommends installing both kernel and userspace tracers: https://github.com/ros2/ros2_tracing#building. Some logic would need to change in the package supporting the ros2 trace command and the Trace action so that it doesn't assume that it's either all (kernel+userspace tracers) or nothing.
  • this has been implemented: https://gitlab.com/ros-tracing/ros2_tracing/-/merge_requests/273

Any other platform-related considerations?

Pull requests

• Include tracepoints by default on Linux ros2_tracing#31
• Add 'lttng-tools' rosdep rule for RHEL ros/rosdistro#36641
• Install LTTng and related packages on Linux ci#690
• Flip ros2_tracing guide to building without instrumentation ros2_documentation#3404
• Add release note about tracepoints being included by default ros2_documentation#3414