Debug Synchronet for *nix using GDB

You can either run Synchronet (sbbs) from the GNU debugger (gdb), or you can debug an sbbs crash “post mortem” provided you have a system-generated core file as a result of a crash.

For the debugger output to be most useful, you need to execute a debug build of Synchronet. That means that the executable files or symlinks in your exec directory should be debug and not release builds. If you run ldd on your exec/sbbs file and it is dependent on libraries in your Synchronet *.lib.release directory, then you are running a release build of sbbs. You need to build withOUT the RELEASE=1 gmake command-line option to build debug binaries and you may need to copy or update the symlinks in your exec directory to use the debug binaries.

Backtraces and other debug output from release builds can still be useful, so if you have a core file from a release build crash, please continue with the debug process using the release build. If more details are needed (e.g. function argument values) to determine the root-cause of the problem, then a similar core from a crash of a debug build will be needed. But it's fine to start with a release build if that's all you have.

Often times, a core file is the best way to find the root cause of a crash, so if you can configure your system to create core files when sbbs crashes, that can be very helpful to the developers in finding and fixing any bugs and ultimately, improving the quality of the software.

You can run ulimit -c to check if core file generation is enabled for the current user profile (0 = disabled, non-zero or “unlimited” = enabled). An “unlimited” core file size is the preferred setting.

On Debian Linux (at least), you can enable unlimited core file generation as the default for all (non-root) users by adding the following line to /etc/security/limits.conf file:

*               soft    core            unlimited

If you're running sbbs daemonzied (e.g. as a *nix service), you may need to edit your service start up script (e.g. /etc/init.d/sbbs) to set the core limit to unlimited:

ulimit -c unlimited

Of in the [Service] section of your /lib/systemd/system/sbbs.service file:

LimitCORE=infinity

Linux Sysops:
To help locate sbbs crash core files, adding the following lines in your /etc/sysctl.conf file can be helpful:

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core.%e.%p

This will place core files with the name core.sbbs.#### in the /tmp directory instead of core.#### in the current directory (typically /sbbs/ctrl).

Also, if you're using the setuid feature of sbbs (e.g. starts as root but changes to a different user after binding ports), then you may need to add the following line to your /etc/sysctl.conf file:

fs.suid_dumpable = 2

You can also set suid_dumpable immediately and temporarily with the following command:

# echo 2 > /proc/sys/fs/suid_dumpable

A. Run the GNU debugger:

# gdb /sbbs/exec/sbbs

or (if debugging with a core file):

# gdb /sbbs/exec/sbbs /tmp/core.sbbs.####

or (if attaching to an existing running instance, reading the PID from /var/run/sbbs.pid):

# gdb /sbbs/exec/sbbs <pid>

or (if attaching to an existing running instance, using pgrep to determine the PID):

# gdb -p $(pgrep sbbs)

B. Run Synchronet non-daemonized (if no core file used):

(gdb) run -nd

C. After segfault or other crash (or when using a core), display back-trace:

(gdb) bt

or (if more details are needed):

(gdb) bt full

D. If (and only if) the last line of the output looks like this:

'#2  0x00000000 in ?? ()' (The number at the start will vary)
display backtraces of all threads:
(gdb) thread apply all bt

E. Copy and paste in e-mail to rob[at]synchro[dot]net or post in one of the Synchronet discussion groups.

TIP:
To run Synchronet (non-daemonized) directly from the GNU debugger without having to type “run” at a (gdb) prompt:

# gdb -ex run --args sbbs -nd

If the sbbs process was started as root, you may need root privileges to attached with gdb:

Attaching to process 21848
ptrace: Operation not permitted.
(gdb)

If you get this error, try attaching again as root (e.g. run with sudo gdb).

When debugging a running sbbs instance, the “broken pipe” signal (SIGPIPE) may normally occur (e.g. when a TCP socket is disconnected) and these signals may cause unwanted breaks into GDB (temporarily stopping sbbs and requiring a g command to continue). To disable this behavior, at the (gdb) prompt, type:

(gdb) handle SIGPIPE nostop noprint pass

Alternatively, you can add the following line to your ~/.gdbinit file or /etc/gdb/gdbinit:

handle SIGPIPE nostop noprint pass

To attach to a running sbbs process and quickly collect a snapshot of backtraces from all running threads (remember, obtain root privileges first, e.g. with sudo, if necessary):

$ gdb -p $(pgrep sbbs) -batch -ex "thread apply all bt" -ex quit > sbbs_threads.txt

• How-To index
• https://wiki.debian.org/HowToGetABacktrace