Last update: 5 August 2020
Copyright © 2014-2020 The LTTng Project
This work is licensed under a Creative Commons Attribution 4.0 International License.
Welcome!
Welcome to the LTTng Documentation!
The Linux Trace Toolkit: next generation is an open source software toolkit which you can use to simultaneously trace the Linux kernel, user applications, and user libraries.
LTTng consists of:
-
Kernel modules to trace the Linux kernel.
-
Shared libraries to trace user applications written in C or C++.
-
Java packages to trace Java applications which use
java.util.loggingor Apache log4j 1.2. -
A Python package to trace Python applications which use the standard
loggingpackage. -
A kernel module to trace shell scripts and other user applications without a dedicated instrumentation mechanism.
-
Daemons and a command-line tool,
lttng, to control the LTTng tracers.
Open source documentation
Note:This is an open documentation: its source is available in a public Git repository.
Should you find any error in the content of this text, any grammatical mistake, or any dead link, we would be very grateful if you would file a GitHub issue for it or, even better, contribute a patch to this documentation by creating a pull request.
Target audience
The material of this documentation is appropriate for intermediate to advanced software developers working in a Linux environment and interested in efficient software tracing. LTTng is also worth a try for students interested in the inner mechanics of their systems.
If you do not have a programming background, you may wish to skip everything related to instrumentation, which often requires at least some programming language skills.
What’s in this documentation?
The LTTng Documentation is divided into the following sections:
-
Nuts and bolts explains the rudiments of software tracing and the rationale behind the LTTng project.
Skip this section if you’re familiar with software tracing and with the LTTng project.
-
Installation describes the steps to install the LTTng packages on common Linux distributions and from their sources.
Skip this section if you already properly installed LTTng on your target system.
-
Quick start is a concise guide to getting started quickly with LTTng kernel and user space tracing.
We recommend this section if you’re new to LTTng or to software tracing in general.
Skip this section if you’re not new to LTTng.
-
Core concepts explains the concepts at the heart of LTTng.
It’s a good idea to become familiar with the core concepts before attempting to use the toolkit.
-
Components of LTTng describes the various components of the LTTng machinery, like the daemons, the libraries, and the command-line interface.
-
Instrumentation shows different ways to instrument user applications and the Linux kernel.
Instrumenting source code is essential to provide a meaningful source of events.
Skip this section if you don’t have a programming background.
-
Tracing control is divided into topics which demonstrate how to use the vast array of features that LTTng 2.12 offers.
-
Reference contains reference tables.
-
Glossary is a specialized dictionary of terms related to LTTng or to the field of software tracing.
Nuts and bolts
What is LTTng? As its name suggests, the Linux Trace Toolkit: next generation is a modern toolkit for tracing Linux systems and applications. So your first question might be: what is tracing?
What is tracing?
As the history of software engineering progressed and led to what we now take for granted—complex, numerous and interdependent software applications running in parallel on sophisticated operating systems like Linux—the authors of such components, software developers, began feeling a natural urge to have tools that would ensure the robustness and good performance of their masterpieces.
One major achievement in this field is, inarguably, the GNU debugger (GDB), an essential tool for developers to find and fix bugs. But even the best debugger won’t help make your software run faster, and nowadays, faster software means either more work done by the same hardware, or cheaper hardware for the same work.
A profiler is often the tool of choice to identify performance bottlenecks. Profiling is suitable to identify where performance is lost in a given software. The profiler outputs a profile, a statistical summary of observed events, which you may use to discover which functions took the most time to execute. However, a profiler won’t report why some identified functions are the bottleneck. Bottlenecks might only occur when specific conditions are met, conditions that are sometimes impossible to capture by a statistical profiler, or impossible to reproduce with an application altered by the overhead of an event-based profiler. For a thorough investigation of software performance issues, a history of execution is essential, with the recorded values of variables and context fields you choose, and with as little influence as possible on the instrumented software. This is where tracing comes in handy.
Tracing is a technique used to understand what goes on in a running software system. The software used for tracing is called a tracer, which is conceptually similar to a tape recorder. When recording, specific instrumentation points placed in the software source code generate events that are saved on a giant tape: a trace file. You can trace user applications and the operating system at the same time, opening the possibility of resolving a wide range of problems that would otherwise be extremely challenging.
Tracing is often compared to logging. However, tracers and loggers are two different tools, serving two different purposes. Tracers are designed to record much lower-level events that occur much more frequently than log messages, often in the range of thousands per second, with very little execution overhead. Logging is more appropriate for a very high-level analysis of less frequent events: user accesses, exceptional conditions (errors and warnings, for example), database transactions, instant messaging communications, and such. Simply put, logging is one of the many use cases that can be satisfied with tracing.
The list of recorded events inside a trace file can be read manually like a log file for the maximum level of detail, but it is generally much more interesting to perform application-specific analyses to produce reduced statistics and graphs that are useful to resolve a given problem. Trace viewers and analyzers are specialized tools designed to do this.
In the end, this is what LTTng is: a powerful, open source set of tools to trace the Linux kernel and user applications at the same time. LTTng is composed of several components actively maintained and developed by its community.
Alternatives to LTTng
Excluding proprietary solutions, a few competing software tracers exist for Linux:
A port of Sun Microsystems’s DTrace to Linux.
The dtrace tool interprets user scripts and is responsible for loading code into the Linux kernel for further execution and collecting the outputted data.
A subsystem in the Linux kernel in which a virtual machine can execute programs passed from the user space to the kernel.
You can attach such programs to tracepoints and kprobes thanks to a system call, and they can output data to the user space when executed thanks to different mechanisms (pipe, VM register values, and eBPF maps, to name a few).
The de facto function tracer of the Linux kernel.
Its user interface is a set of special files in sysfs.
A performance analysis tool for Linux which supports hardware performance counters, tracepoints, as well as other counters and types of probes.
perf’s controlling utility is the perf command line/text UI tool.
A command-line utility which records system calls made by a user process, as well as signal deliveries and changes of process state.
strace makes use of ptrace to fulfill its function.
Like SystemTap, uses scripts to analyze Linux kernel events.
You write scripts, or chisels in sysdig’s jargon, in Lua and sysdig executes them while it traces the system or afterwards. sysdig’s interface is the sysdig command-line tool as well as the text UI-based csysdig tool.
A Linux kernel and user space tracer which uses custom user scripts to produce plain text traces.
SystemTap converts the scripts to the C language, and then compiles them as Linux kernel modules which are loaded to produce trace data. SystemTap’s primary user interface is the stap command-line tool.
The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, a file format optimized for the production and analyses of multi-gigabyte data.
LTTng is the result of more than 10 years of active open source development by a community of passionate developers. LTTng is currently available on major desktop and server Linux distributions.
The main interface for tracing control is a single command-line tool named lttng. The latter can create several tracing sessions, enable and disable events on the fly, filter events efficiently with custom user expressions, start and stop tracing, and much more. LTTng can record the traces on the file system or send them over the network, and keep them totally or partially. You can view the traces once tracing becomes inactive or in real-time.
Install LTTng now and start tracing!
Installation
LTTng is a set of software components which interact to instrument the Linux kernel and user applications, and to control tracing (start and stop tracing, enable and disable event rules, and the rest). Those components are bundled into the following packages:
LTTng-tools
Libraries and command-line interface to control tracing.
LTTng-modules
Linux kernel modules to instrument and trace the kernel.
LTTng-UST
Libraries and Java/Python packages to instrument and trace user applications.
Most distributions mark the LTTng-modules and LTTng-UST packages as optional when installing LTTng-tools (which is always required). In the following sections, we always provide the steps to install all three, but note that:
-
You only need to install LTTng-modules if you intend to trace the Linux kernel.
-
You only need to install LTTng-UST if you intend to trace user applications.
Availability of LTTng 2.12 for major Linux distributions as of 5 August 2020.
| Distribution | Available in releases |
|---|---|
| Ubuntu 16.04 Xenial Xerus, Ubuntu 18.04 Bionic Beaver, and Ubuntu 20.04 Focal Fossa: use the LTTng Stable 2.12 PPA. |
|
Quick start
This is a short guide to get started quickly with LTTng kernel and user space tracing.
Before you follow this guide, make sure to install LTTng.
This tutorial walks you through the steps to:
-
Trace a user application written in C.
Core concepts
From a user’s perspective, the LTTng system is built on a few concepts, or objects, on which the lttng command-line tool operates by sending commands to the session daemon. Understanding how those objects relate to eachother is key in mastering the toolkit.
The core concepts are:
Components of LTTng
The second T in LTTng stands for toolkit: it would be wrong to call LTTng a simple tool since it is composed of multiple interacting components. This section describes those components, explains their respective roles, and shows how they connect together to form the LTTng ecosystem.
The following diagram shows how the most important components of LTTng interact with user applications, the Linux kernel, and you:
Instrumentation
There are many examples of tracing and monitoring in our everyday life:
-
You have access to real-time and historical weather reports and forecasts thanks to weather stations installed around the country.
-
You know your heart is safe thanks to an electrocardiogram.
-
You make sure not to drive your car too fast and to have enough fuel to reach your destination thanks to gauges visible on your dashboard.
All the previous examples have something in common: they rely on instruments. Without the electrodes attached to the surface of your body’s skin, cardiac monitoring is futile.
LTTng, as a tracer, is no different from those real life examples. If you’re about to trace a software system or, in other words, record its history of execution, you better have instrumentation points in the subject you’re tracing, that is, the actual software.
Various ways were developed to instrument a piece of software for LTTng tracing. The most straightforward one is to manually place instrumentation points, called tracepoints, in the software’s source code. It is also possible to add instrumentation points dynamically in the Linux kernel tracing domain.
If you’re only interested in tracing the Linux kernel, your instrumentation needs are probably already covered by LTTng’s built-in Linux kernel tracepoints. You may also wish to trace a user application which is already instrumented for LTTng tracing. In such cases, skip this whole section and read the topics of the Tracing control section.
Many methods are available to instrument a piece of software for LTTng tracing. They are:
Tracing control
Once an application or a Linux kernel is instrumented for LTTng tracing, you can trace it.
This section is divided in topics on how to use the various components of LTTng, in particular the lttng command-line tool, to control the LTTng daemons and tracers.
Note:In the following subsections, we refer to an lttng(1) command using its man page name. For example, instead of Run the create command to…, we use Run the lttng-create(1) command to….
When you create a tracing session, it is set as the current tracing session. The following lttng(1) commands operate on the current tracing session when you don’t specify one: