strace and ltrace: Debugging Linux Programs You Didn't Write

Overview

A binary from a vendor fails on the production server and works on your laptop. There's no source, no logs, and no documentation. This is where strace earns its place — it shows every system call the program makes, so you can see exactly what it's trying to do and where it's failing.

It's the closest thing Linux has to reading a program's mind.

What strace actually shows you

Every interaction between a program and the kernel is a system call. Opening a file, reading from a socket, allocating memory, forking a child process — all of it goes through syscalls, and strace logs them.

strace -f ./myprogram

The -f flag follows child processes, which you almost always want. Without it, you see the parent's syscalls but miss everything the children do, and most real programs spawn children.

Output looks like this:

openat(AT_FDCWD, "/etc/myapp/config.yml", O_RDONLY) = -1 ENOENT (No such file)
openat(AT_FDCWD, "/usr/local/etc/myapp/config.yml", O_RDONLY) = 3
read(3, "port: 8080\nhost: 0.0.0.0\n", 4096) = 25
...
connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.0.5")}, 16) = -1 ECONNREFUSED (Connection refused)

Even from those four lines you can see: the config it couldn't find, the one it did find, and the database connection that failed. That's the entire bug, visible in under a second.

The flags that matter

Flag	Effect
`-f`	Follow child processes
`-e trace=openat,open,read`	Only show specific syscalls
`-p PID`	Attach to a running process
`-o file.txt`	Write output to a file
`-s 256`	Show more bytes of string arguments (default 32)
`-tt`	Timestamps with microseconds
`-T`	Time spent in each syscall
`-c`	Summary of syscall counts at exit
`-y`	Show paths associated with file descriptors

-y is the one people don't know about and should. It annotates file descriptors with what they point to, so instead of seeing read(3, ...) you see read(3</etc/myapp/config.yml>, ...). Makes complex output readable.

Reading strace output without drowning

Real programs make thousands of syscalls. The trick is filtering.

Find a missing file

strace -f -e trace=openat -s 256 ./myprogram 2>&1 | grep ENOENT

Every file the program tried to open and couldn't. This is how you find the config file it's looking for, or the shared library that's not installed.

Find a connection problem

strace -f -e trace=connect ./myprogram 2>&1 | grep -v "= 0"

Every connect call that didn't succeed. Shows the address it tried and the error code.

Find what a hung process is doing

strace -f -p $(pgrep myprogram) -e trace=network,read,write

Attach to a live process and watch. If nothing is happening, the process is either blocked on a syscall (you'll see it, unfinished) or busy in userspace (you won't). The distinction is immediately actionable.

Get a profile of syscall usage

strace -c -f ./myprogram

Shows counts and time per syscall. Occasionally reveals surprising things — a program making 50,000 stat calls is probably doing something inefficient.

ltrace: same idea, library calls

strace shows kernel interactions. ltrace shows library calls — malloc, strcpy, and any function in a shared library the program calls. This is closer to the program's logic.

ltrace ./myprogram

Output is messier because there are far more library calls than syscalls. In practice I use ltrace far less than strace, mostly when I need to see what arguments a specific library function is getting.

One limitation: ltrace only works on dynamically linked binaries. Statically linked Go binaries, for instance, won't show anything useful because all the library code is compiled in.

A real debugging session

This is a problem I actually had: a third-party CLI tool that read credentials from a config file, but on one server it kept saying "authentication failed" despite the config being present.

strace -f -e trace=openat,read -s 256 /usr/local/bin/vendor-tool deploy 2>&1 | head -50

The output showed it opening /etc/vendor-tool/credentials, reading it, and then later opening /root/.vendor-tool/credentials and failing. The second one didn't exist.

The tool was checking two locations. The first file existed but was empty because a config management tool had created it without writing contents. The tool read it, found nothing, moved on to the second path, and failed.

Without strace, I'd have been checking the one config file I knew about, which looked correct. The problem was a second file I didn't know existed. Five minutes with strace versus an afternoon of guessing.

Security notes

Two things to be aware of when using strace:

Output can contain secrets. If a program reads a password from a file or environment variable, it appears in the strace output. Don't paste strace logs into tickets without checking.
It's slow. strace intercepts every syscall, which can slow a program by 10x or more. Never run it against a production process you care about without testing on staging first.

Also: on some systems, ptrace is restricted. If you get "ptrace: Operation not permitted," check /proc/sys/kernel/yama/ptrace_scope. Setting it to 0 allows attaching to any process you own; the default of 1 restricts it to children.

When to reach for strace

A binary fails with a vague error and no logs
A program works as one user but not another (permission issues show up as EACCES in strace)
A service hangs and you don't know why
You need to know what network endpoints a program talks to
Something is slow and you suspect excessive filesystem access

It's not the first tool to reach for. Check logs, check config, check permissions first. But when those don't explain it, strace does, and it does it in seconds.