= bpftrace(8)
:doctype: manpage
:toc: true
:toclevels: 1

////
Style guide:
- one sentence per line
////

== Name

bpftrace - a high-level tracing language

== Synopsis

*bpftrace* [_OPTIONS_] _FILENAME_ +
*bpftrace* [_OPTIONS_] -e 'program code'

When _FILENAME_ is "_-_", bpftrace will read program code from stdin.

A program will continue running until Ctrl-C is hit, or an `exit` function is called.
When a program exits, all populated maps are printed (more details below).

== Description

bpftrace is a high-level tracing language for Linux. bpftrace uses LLVM as
a backend to compile scripts to eBPF-bytecode and makes use of libbpf and bcc
for interacting with the Linux BPF subsystem, as well as existing Linux
tracing capabilities.

== Examples

Trace processes calling sleep::
----
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
----

Trace processes calling sleep while spawning `sleep 5` as a child process::
----
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }' -c 'sleep 5'
----

List all probes with "sleep" in their name::
----
# bpftrace -l '*sleep*'
----

List all the probes attached in the program::
----
# bpftrace -l -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
----

== Options

=== *-B* _MODE_

Set the output buffer mode (applies to terminal and file output `-o`).

Valid values are::
*none* No buffering. Each I/O is written as soon as possible +
*line* Data is written on the first newline or when the buffer is full. This is the default mode. +
*full* Data is written once the buffer is full.

=== *-c* _COMMAND_

Run _COMMAND_ as a child process.
When the child terminates bpftrace will also terminate, as if 'exit()' had been called.
If bpftrace terminates before the child process does the child process will be terminated with a SIGTERM.
If used, 'USDT' probes will only be attached to the child process.
To avoid a race condition when using 'USDTs', the child is stopped after 'execve' using 'ptrace(2)' and continued when all 'USDT' probes are attached.
The child process runs with the same privileges as bpftrace itself (usually root).

Unless otherwise specified, bpftrace does not perform any implicit filtering. Therefore, if you are only interested in
events in _COMMAND_, you may want to filter based on the child PID. The child PID is available to programs as the 'cpid' builtin.
For example, you could add the predicate `/pid == cpid/` to probes with userspace context.

=== *-d STAGE*

Enable debug mode.
For more details see the <<Debug Output>> section.

=== *--debuginfo* _DIR[:DIR]_

Add the directory DIR to the search path for DWARF debug information.
Paths may be absolute or relative to the traced binary's location.
Specify multiple paths either by separating them with a colon (`:`) or by repeating the option,
e.g. `--debuginfo=/bin/debug:./lib/debug:..` and `--debuginfo=/bin/debug --debuginfo=./lib/debug`, respectively.
Debug files in these paths are matched by build ID when available; prefix a path with `+` to enable CRC32 validation when needed.
Files that do not match are skipped.
By default, bpftrace searches under standard debug paths, including `.debug` relative to the traced binary and `/usr/lib/debug`; it may also query available debuginfod servers.

Split debuginfo format (`.dwo`, `.dwp`) is not yet supported for this option, but continues to work with the default probe binary path.

=== *--dry-run*

Terminate execution right after attaching all the probes. Useful for testing
that the script can be parsed, loaded, and attached, without actually running
it.

=== *-e* _PROGRAM_

Execute _PROGRAM_ instead of reading the program from a file or stdin.

=== *-f* _FORMAT_

Set the output format.

Valid values are::
*json* +
*text*

The JSON output is compatible with NDJSON and JSON Lines, meaning each line of the streamed output is a single blob of valid JSON.

=== *--fmt* _FILENAME_

Output standard format for the bpftrace file _FILENAME_.

=== *-h, --help*

Print the help summary.

=== *-I* _DIR_

Add the directory _DIR_ to the search path for C headers.
This option can be used multiple times.
For more details see the <<Preprocessor Options>> section.

=== *--include* _FILENAME_

Add _FILENAME_ as an include for the pre-processor.
This is equal to adding '#include _FILENAME_' at the top of the program.
This option can be used multiple times.
For more details see the <<Preprocessor Options>> section.

=== *--info*

Print detailed information about features supported by the kernel and the bpftrace build.

=== *-k*

This flag enables runtime warnings for errors from 'probe_read_*' and map lookup BPF helpers.
When errors occur bpftrace will log an error containing the source location and the error code:

----
stdin:48-57: WARNING: Failed to probe_read_user_str: Bad address (-14)
u:lib.so:"fn(char const*)" { printf("arg0:%s\n", str(arg0));}
                                                 ~~~~~~~~~
----

=== *-l* [_SEARCH_|_FILENAME_]

List all probes that match the _SEARCH_ pattern.
If the pattern is omitted all probes will be listed.
This pattern supports wildcards in the same way that probes do.
E.g. '-l kprobe:*file*' to list all 'kprobes' with 'file' in the name.
This can be used with a program, which will list all probes in that program.
For more details see the <<Listing Probes>> section.

=== *--no-feature* _feature,feature,..._

Disable use of detected features, valid values are::
*uprobe_multi* to disable uprobe_multi link +
*kprobe_multi* to disable kprobe_multi link +
*kprobe_session* to disable automatic collapse of kprobe/kretprobe into kprobe session

=== *--no-warnings*

Suppress all warning messages created by bpftrace.

=== *--probe-filter* _REGEX_

Only attach and run probes whose name matches the regular expression _REGEX_.
Probes that do not match are not loaded into the kernel.
Special probes (`BEGIN` and `END`) are not affected by this filter.
If no probes match, bpftrace exits with an error.

=== *--traceable-functions* _FILENAME_

Specify the file containing the list of traceable kernel functions. If not set,
bpftrace uses `/sys/kernel/tracing/available_filter_functions`, which depends on
dynamic ftrace. The format is the same as in `available_filter_functions`: each
line is either "symbol" or "symbol [module]".

=== *-o* _FILENAME_

Write bpftrace tracing output to _FILENAME_ instead of stdout.
This doesn't include child process (*-c* option) output.
Errors are still written to stderr.

=== *-p* _PID_

Attach to the process with or filter actions by _PID_.
If the process terminates, bpftrace will also terminate.
When using USDT, uprobes, uretprobes, hardware, software, profile, interval, or watchpoint probes they will be attached to only this process.
For all other probes, except begin/end, the pid will act like a predicate to filter out events not from that pid.
For listing uprobes/uretprobes set the target to '*' and the process's address space will be searched for the symbols.

=== *-q*

Keep messages quiet.

=== *--unsafe*

Some calls, like 'system', are marked as unsafe as they can have dangerous side effects ('system("rm -rf")') and are disabled by default.
This flag allows their use.

In addition, it makes kprobe/kretprobe checks less restrictive.
It can be used to probe functions that bpftrace reports as not traceable, but are supported if the kernel is configured to allow probing notrace functions.

=== *--usdt-file-activation*

Activate usdt semaphores based on file path.

=== *-V, --version*

Print bpftrace version information.

=== *-v*

Enable verbose messages.
For more details see the <<Verbose Output>> section.

=== Program Options

You can also pass custom options to a bpftrace program/script itself via positional or named parameters.
Positional parameters can be placed before or after a double dash but named parameters can ONLY come after a double dash; e.g.
```
# bpftrace -e 'begin { print(($1, $2, getopt("aa", 1), getopt("bb"))); }' p1 -- --aa=20 --bb p2

// (p1, p2, 20, true) is printed
```
or
```
# bpftrace myscript.bt -- p1 --aa=20 --bb p2
```

In these examples there are two positional parameters (`p1`, `p2`) and two named parameters (`aa`, which is set to `20`, and `bb`, which is set to `true`).
Named program parameters require the `=` to set their value unless they are boolean parameters (like 'bb' above).
Read about how to access positional and named parameters link:https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#command-line-parameters[here].

== The Language

Syntax, types, and concepts for bpftrace are link:https://github.com/bpftrace/bpftrace/blob/master/docs/language.md[available here].

== Probes

bpftrace supports various probe types which allow the user to attach BPF programs to different types of events.
Each probe starts with a provider (e.g. `kprobe`) followed by a colon (`:`) separated list of options.
The amount of options and their meaning depend on the provider.
Optionally, the attachpoint specified can be prefixed by a user-provided name followed by equal sign (e.g. `myname=kprobe:...`); this is used for certain advanced features and is typically omitted.
link:https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#probes[Full list of probe types].

== Standard library

The standard library of all link:https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#builtins[builtins], link:https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#functions[functions], and link:https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md#map-functions[map functions] is link:https://github.com/bpftrace/bpftrace/blob/master/docs/stdlib.md[available here].

== Configuration

=== Config Variables

Some behavior can only be controlled through config variables, which are link:https://github.com/bpftrace/bpftrace/blob/master/docs/language.md#config-variables[available here].
These can be set via the Config Block directly in a script (before any probes) or via their environment variable equivalent, which is upper case and includes the `BPFTRACE_` prefix e.g. ``stack_mode``'s environment variable would be `BPFTRACE_STACK_MODE`.

=== Environment Variables

These are not available as part of the standard set of Config Variables above and can only be set as environment variables.

==== BPFTRACE_BTF

Default: None

The path to a BTF file. By default, bpftrace searches several locations to find a BTF file.
See src/btf.cpp for the details.

==== BPFTRACE_KERNEL_BUILD

Default: `/lib/modules/$(uname -r)`

Only used with `BPFTRACE_KERNEL_SOURCE` if it is out-of-tree Linux kernel build.

==== BPFTRACE_KERNEL_SOURCE

Default: `/lib/modules/$(uname -r)`

bpftrace requires kernel headers for certain features, which are searched for in this directory.

==== BPFTRACE_VMLINUX

Default: None

This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset.
If this value is not given, bpftrace searches vmlinux from pre defined locations.
See src/attached_probe.cpp:find_vmlinux() for details.

==== BPFTRACE_COLOR

Default: auto

Colorize the bpftrace log output message. Valid values are auto, always and never.

== Options Expanded

=== Debug Output

The `-d STAGE` option produces debug output. It prints the output of the
bpftrace execution stage given by the _STAGE_ argument. The option can be used
multiple times (with different stage names) and the special value `all` prints
the output of all the supported stages. The option also takes multiple stages
in one invocation as comma separated values.

Note: This is primarily used for bpftrace developers.

The supported options are:

[cols="~,~"]
|===

| `ast`
| Prints the Abstract Syntax Tree (AST) after every pass.

| `codegen`
| Prints the unoptimized LLVM IR as produced by `CodegenLLVM`.

| `codegen-opt`
| Prints the optimized LLVM IR, i.e. the code which will be compiled into BPF
bytecode.

| `dis`
| Disassembles and prints out the generated bytecode that `libbpf` will see.
Only available in debug builds.

| `libbpf`
| Captures and prints libbpf log for all libbpf operations that bpftrace uses.

| `verifier`
| Captures and prints the BPF verifier log.

| `all`
| Prints the output of all of the above stages.

|===

=== Listing Probes

Probe listing is the method to discover which probes are supported by the current system.
Listing supports the same syntax as normal attachment does and alternatively can be
combined with `-e` or filename args to see all the probes that a program would attach to.

----
# bpftrace -l 'kprobe:*'
# bpftrace -l 't:syscalls:*openat*
# bpftrace -l 'kprobe:tcp*,trace
# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
# bpftrace -l -e 'tracepoint:xdp:mem_* { exit(); }'
# bpftrace -l my_script.bt
# bpftrace -lv 'enum cpu_usage_stat'
----

The verbose flag (`-v`) can be specified to inspect arguments (`args`) for providers that support it:

----
# bpftrace -l 'fexit:tcp_reset,tracepoint:syscalls:sys_enter_openat' -v
fexit:tcp_reset
    struct sock * sk
    struct sk_buff * skb
tracepoint:syscalls:sys_enter_openat
    int __syscall_nr
    int dfd
    const char * filename
    int flags
    umode_t mode

# bpftrace -l 'uprobe:/bin/bash:rl_set_prompt' -v    # works only if /bin/bash has DWARF
uprobe:/bin/bash:rl_set_prompt
    const char *prompt

# bpftrace -lv 'struct css_task_iter'
struct css_task_iter {
        struct cgroup_subsys *ss;
        unsigned int flags;
        struct list_head *cset_pos;
        struct list_head *cset_head;
        struct list_head *tcset_pos;
        struct list_head *tcset_head;
        struct list_head *task_pos;
        struct list_head *cur_tasks_head;
        struct css_set *cur_cset;
        struct css_set *cur_dcset;
        struct task_struct *cur_task;
        struct list_head iters_node;
};
----

=== Preprocessor Options

The `-I` option can be used to add directories to the list of directories that bpftrace uses to look for headers.
Can be defined multiple times.

----
# cat program.bt
#include <foo.h>

begin { @ = FOO }

# bpftrace program.bt

definitions.h:1:10: fatal error: 'foo.h' file not found

# /tmp/include
foo.h

# bpftrace -I /tmp/include program.bt

Attached 1 probe
----

The `--include` option can be used to include headers by default.
Can be defined multiple times.
Headers are included in the order they are defined, and they are included before any other include in the program being executed.

----
# bpftrace --include linux/path.h --include linux/dcache.h \
    -e 'kprobe:vfs_open { printf("open path: %s\n", str(((struct path *)arg0).dentry.d_name.name)); }'

Attached 1 probe
open path: .com.google.Chrome.ASsbu2
open path: .com.google.Chrome.gimc10
open path: .com.google.Chrome.R1234s
----

=== Verbose Output

The `-v` option prints more information about the program as it is run:

----
# bpftrace -v -e 'tracepoint:syscalls:sys_enter_nanosleep { printf("%s is sleeping.\n", comm); }'
AST node count: 7
Attached 1 probe

load tracepoint:syscalls:sys_enter_nanosleep, with BTF, with func_infos: Success

Program ID: 111
Attaching tracepoint:syscalls:sys_enter_nanosleep
iscsid is sleeping.
iscsid is sleeping.
[...]
----

== Terminology

[cols="~,~"]
|===

| BPF
| Berkeley Packet Filter: a kernel technology originally developed for optimizing the processing of packet filters (eg, tcpdump expressions).

| BPF map
| A BPF memory object, which is used by bpftrace to create many higher-level objects.

| BTF
| BPF Type Format: the metadata format which encodes the debug info related to BPF program/map.

| dynamic tracing
| Also known as dynamic instrumentation, this is a technology that can instrument any software event, such as function calls and returns, by live modification of instruction text. Target software usually does not need special capabilities to support dynamic tracing, other than a symbol table that bpftrace can read. Since this instruments all software text, it is not considered a stable API, and the target functions may not be documented outside of their source code.

| eBPF
| Enhanced BPF: a kernel technology that extends BPF so that it can execute more generic programs on any events, such as the bpftrace programs listed below. It makes use of the BPF sandboxed virtual machine environment. Also note that eBPF is often just referred to as BPF.

| kprobes
| A Linux kernel technology for providing dynamic tracing of kernel functions.

| probe
| An instrumentation point in software or hardware, that generates events that can execute bpftrace programs.

| static tracing
| Hard-coded instrumentation points in code. Since these are fixed, they may be provided as part of a stable API, and documented.

| tracepoints
| A Linux kernel technology for providing static tracing.

| uprobes
| A Linux kernel technology for providing dynamic tracing of user-level functions.

| USDT
| User Statically-Defined Tracing: static tracing points for user-level software. Some applications support USDT.

|===

== Program Files

Programs saved as files are often called scripts and can be executed by specifying their file name.
It is convention to use the `.bt` file extension but it is not required.

For example, listing the sleepers.bt file using `cat`:

----
# cat sleepers.bt

tracepoint:syscalls:sys_enter_nanosleep {
  printf("%s is sleeping.\n", comm);
}
----

And then calling it:

----
# bpftrace sleepers.bt

Attached 1 probe
iscsid is sleeping.
iscsid is sleeping.
----

It can also be made executable to run stand-alone.
Start by adding an interpreter line at the top (`#!`) with either the path to your installed bpftrace (/usr/local/bin is the default) or the path to `env` (usually just `/usr/bin/env`) followed by `bpftrace` (so it will find bpftrace in your `$PATH`):

----
#!/usr/local/bin/bpftrace

tracepoint:syscalls:sys_enter_nanosleep {
  printf("%s is sleeping.\n", comm);
}
----

Then make it executable:

----
# chmod 755 sleepers.bt
# ./sleepers.bt

Attached 1 probe
iscsid is sleeping.
iscsid is sleeping.
----
