Troubleshooting SUID programs

Today I was trying to resolve an issue with PulseAudio on my Fedora 9 system. For some reason, pulseaudio would not acquire real-time privileges. I had already grepped through the logs, which gave me a solution direction but not a solution. The next step I wanted to try is to run pulseaudio under a debugger.

But .. pulseaudio is SUID root. It requires this so that it can start up its I/O threads with real-time priority. Unfortunately, this also means that you cannot debug (or strace, both use the ptrace() system call) the process as an unprivileged user.

My first attempt around this was to run the debugger as root, as below.

# gdb pulseaudio

However, this has the problem that you will also run pulseaudio as root. SUID applications are intended to be run as an unprivileged user (otherwise they would not be SUID) so you can expect erratic behaviour. My problem was finding out why PulseAudio didn't start up with real-time privileges. If I just ran the program as root chances are that this would mask the problem.

So my next thought was to use sudo to run pulseaudio under my normal user id, through the debugger (edited for brevity).

# gdb sudo
(gdb) set args -u gjansen -H pulseaudio

However I wanted to set a breakpoint in pulseaudio's main() function which is not possible this way. Issuing the "b main" command in gdb would set a breakpoint in sudo's main() function, not in that of pulseaudio!

After thinking of the problem for a bit more, I concluded that you really need to run the program you're troubleshooting under your own UID, and attach the debugger as root from another session.

However, again a problem arises: the problem that I was troubleshooting was in the initialization code in pulseaudio. I would never be able to start up pulseaudio, switch to another console, find the PID, and attach the debugger in time. I really needed a way to start pulseaudio and pause it immediately. You can try this by pressing CTRL-Z immediately after you start it. However, you need to be extremely quick and the approach still seemed clunky.

So I decided to solve the problem properly. I created a small shared library that you can LD_PRELOAD with an SUID executable and which stops the program before it reaches its main() function. How does it work? It is actually quite simple. ELF files have a .init section that contains code which is executed by the dynamic linker before it transfers control to the program. This section is used e.g. for executing the contructors of global objects in C++. With some gcc magic, you can add your own function to this section. This is what my library does: it add a small function to the .init section that sends a SIGSTOP to itself. It will also output the process ID, as a convenience.

The code is given below:

#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <sys/types.h>

static void __attribute__((constructor))
stop_init_func()
{
    pid_t pid = getpid();
    printf("Sending SIGSTOP to myself (pid = %d)\n", pid);
    kill(pid, SIGSTOP);
}

To compile this code in a shared library, save the code above a file called "stop.c" and issue the following commands:

$ gcc -fPIC -c -o stop.o stop.c
$ gcc -shared -o stop.so stop.o

The library needs to be installed in /usr/lib (or /usr/lib64 on bi-arch systems), and needs to be SUID root. The latter is required to allow it to be preloaded into SUID programs.

# cp stop.so /usr/lib
# chmod u+s /usr/lib/stop.so

Now let's show how to use this library with a simple SUID "hello world" program.

$ ls -l hello
-rwsr-xr-x 1 root root 8225 2008-06-14 14:17 hello
$ LD_PRELOAD=stop.so ./hello
Sending SIGSTOP to myself (pid = 32269)

[1]+  Stopped                 LD_PRELOAD=stop.so ./hello
$

As you can see the program printed its process ID and then stopped itself. Now you can attach gdb as root in another session as displayed below (edited for brevity).

# gdb hello 32269
Attaching to program: /home/gjansen/Scratch/link/hello, process 32269
Redelivering pending Stopped (signal).
Reading symbols from /usr/lib64/stop.so...done.
Loaded symbols for /usr/lib64/stop.so
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x0000003573232507 in kill () from /lib64/libc.so.6
(gdb)

Voila! The program has not yet executed its main() function, so if you want you can set a breakpoint there or anywhere else. Then enter "c" to continue the program.

UPDATE 2008/08/23: The source code for stop.c as well as a Makefile can be downloaded from here (thanks to the guys at freeHg).