====== 🐛 Debug Synchronet for *nix using GDB ======

You can either run Synchronet (''sbbs'') from the GNU debugger (''gdb''), or you
can debug an sbbs crash "post mortem" provided you have a system-generated
core file as a result of a crash.

===== Debug Build =====

For the debugger output to be most useful, you need to execute a //debug// build of Synchronet. That means that the executable files or symlinks in your ''[[dir:exec]]'' directory should be //debug// and not //release// builds. If you run ''ldd'' on your ''[[dir:exec]]/sbbs'' file and it is dependent on libraries in your Synchronet ''*.lib.release'' directory, then you are running a //release// build of sbbs. You need to build withOUT the ''RELEASE=1'' gmake command-line option to build debug binaries and you may need to copy or update the symlinks in your ''[[dir:exec]]'' directory to use the //debug// binaries.

Backtraces and other debug output from //release// builds can still be useful, so if you have a core file from a //release// build crash, please continue with the debug process using the //release// build. If more details are needed  (e.g. function argument values) to determine the root-cause of the problem, then a similar core from a crash of a //debug// build will be needed. But it's fine to start with a //release// build if that's all you have.
===== Core File =====

Often times, a core file is the best way
to find the root cause of a crash, so if you can configure your system to
create core files when sbbs crashes, that can be very helpful to the
developers in finding and fixing any bugs and ultimately, improving the
quality of the software.

You can run ''ulimit -c'' to check if core file generation is enabled for the
current user profile (0 = disabled, non-zero or "unlimited" = enabled). An 
"unlimited" core file size is the preferred setting.

On Debian Linux (at least), you can enable unlimited core file generation as the default for
all (non-root) users by adding the following line to ''/etc/security/limits.conf'' file:
<file>
*               soft    core            unlimited
</file>

If you're running sbbs daemonzied (e.g. as a *nix service), you may need to edit your service
start up script (e.g. ''/etc/init.d/sbbs'') to set the core limit to //unlimited//:
  ulimit -c unlimited
  
Of in the ''[Service]'' section of your ''/lib/systemd/system/sbbs.service'' file:
  LimitCORE=infinity
  
:!: **Linux Sysops**:\\
To help locate sbbs crash core files, adding the following lines in your ''/etc/sysctl.conf'' or ''/etc/sysctl.d/sysctl.local.conf'' file can be helpful:

  # Controls whether core dumps will append the PID to the core filename.
  # Useful for debugging multi-threaded applications.
  kernel.core_uses_pid = 1
  kernel.core_pattern = /tmp/core.%e.%p

This will place core files with the name ''core.sbbs.####'' in the ''/tmp'' directory
instead of ''core.####'' in the current directory (typically ''/sbbs/ctrl'').
To reload the modified sysctl configuration files, run
  $ /sbin/sysctl --system

Also, if you're using the ''setuid'' feature of sbbs (e.g. starts as //root// but changes to a different user after binding ports), then you may need to add the following line to your ''/etc/sysctl.conf'' file:
  fs.suid_dumpable = 2
  
You can also set ''suid_dumpable'' immediately and temporarily with the following command:
  $ echo 2 > /proc/sys/fs/suid_dumpable
  

===== Debugging =====

A. Run the GNU debugger:
  # gdb /sbbs/exec/sbbs
or (if debugging with a core file):
  # gdb /sbbs/exec/sbbs /tmp/core.sbbs.####
or (if attaching to an existing running instance, reading the PID from ''/var/run/sbbs.pid''):
  # gdb /sbbs/exec/sbbs <pid>
or (if attaching to an existing running instance, using ''pidof'' to determine the PID):
  # gdb -p $(pidof sbbs)

B. Run Synchronet non-daemonized (if no core file used):
  (gdb) run -nd

C. After segfault or other crash (or when using a core), display back-trace:
  (gdb) bt
or (if more details are needed):
  (gdb) bt full

D. If (and only if) the last line of the output looks like this:
  '#2  0x00000000 in ?? ()' (The number at the start will vary)
  display backtraces of all threads:
  (gdb) thread apply all bt

E. Copy and paste in e-mail to rob[at]synchro[dot]net or post in one of the Synchronet [[support|discussion groups]].

**TIP:**\\
To run Synchronet (non-daemonized) directly from the GNU debugger without having to type "run" at a ''(gdb)'' prompt:
  # gdb -ex run --args sbbs -nd

==== Root Access ====

If the ''sbbs'' process was started as root, you may need root privileges to attached with ''gdb'':

  Attaching to process 21848
  ptrace: Operation not permitted.
  (gdb)
  
If you get this error, try attaching again as root (e.g. run with ''sudo gdb'').

==== Handling SIGPIPE ====

When debugging a //running// ''sbbs'' instance, the "broken pipe" signal (''SIGPIPE'') may normally occur (e.g. when a TCP socket is disconnected) and these signals may cause unwanted breaks into GDB (temporarily stopping sbbs and requiring a ''g'' command to continue). To disable this behavior, at the ''(gdb)'' prompt, type:
  (gdb) handle SIGPIPE nostop noprint pass
  
Alternatively, you can add the following line to your ''~/.gdbinit'' file or ''/etc/gdb/gdbinit'':
  handle SIGPIPE nostop noprint pass
  
===== Thread Snapshot =====

To attach to a running sbbs process and quickly collect a snapshot of backtraces from all running threads (remember, obtain root privileges first, e.g. with ''sudo'', if necessary):
  $ gdb -p $(pidof sbbs) -batch -ex "thread apply all bt" -ex quit > sbbs_threads.txt

===== See Also =====
  * [[:howto:|How-To index]]
  * [[https://wiki.debian.org/HowToGetABacktrace]]

{{tag>debug unix linux gdb}}