How Ubuntu dumps core

When your program tries to do anything nasty, like access memory it is not supposed to, you end up seeing a core dump.

If you ran the program under the Bash shell, you might see this error:

$ ./my_program
Segmentation fault (core dumped)

Under the Fish shell, I see this error:

$ ./my_program
fish: β€œ./my_program” terminated by signal SIGSEGV (Address boundary error)

If I check the directory I ran the program from, I see that there is a core file that has the entire dump of the process memory from when the program did its nasty act.

How did a SIGSEGV end up as a core dump file? This is what I found in Ubuntu:

  • On SIGSEGV, Linux kernel writes an entry in /var/log/syslog with some information about what happened. Here is the line on my computer:
Aug 13 17:56:09 my_machine kernel: [442853.259571] my_program[18432]: segfault at 4700000046 ip 00007f7688f6980d sp 00007ffd46b5e3e0 error 4 in libc-2.23.so[7f7688f30000+1c0000]
  • The Linux kernel checks what settings are present for writing a core dump. There are 3 settings, which are in the files core_pattern, core_pipe_limit and core_uses_pid in /proc/sys/kernel directory. The documentation of these settings can be found described in kernel.txt.

  • core_pattern says what the filename of the core dump file should be. On my Ubuntu, this setting had this value:

$ cat /proc/sys/kernel/core_pattern 
|/usr/share/apport/apport %p %s %c %P

From the kernel.txt documentation, we can see that this means that the kernel has to call the /usr/share/apport/apport executable with four input arguments. The first, second and fourth arguments are documented as PID of offending process, signal number and global PID. We do not know what the %c does, but we will find out soon below. The kernel will pipe the process memory to the standard input of this apport program.

  • What is this apport program? It is the crash handler of Ubuntu. We can see that it is a Python script:
$ file /usr/share/apport/apport
/usr/share/apport/apport: Python script, ASCII text executable

That is great, because we can open it and read its code to see what it does!

  • Opening up this file in Vim, we see that %c input argument is the ulimit value set in our shell:
Line 393: (pid, signum, core_ulimit) = sys.argv[1:4]
  • We can also see what exactly it does to write the core dump in its function write_user_coredump. In short, this function checks the ulimit value of the shell and then decides how many bytes to write to the core file and writes it to the same directory as the program.

  • We also see that it called error_log to write some log messages. These can be found listed in the /var/log/apport.log file:

ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: called for pid 18474, signal 11, core limit 102400000
ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: executable: /home/joe/my_program (command line "./my_program")
ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: executable does not belong to a package, ignoring
ERROR: apport (pid 18475) Sat Aug 13 17:56:18 2016: writing core dump to /home/joe/core (limit: 102400000)

This is how Ubuntu writes the core dump file. You can now open this up in GDB for debugging! πŸ™‚

Tried with: Ubuntu 16.04

How to load core dump in GDB

Running an erroneous program may result it in exiting and dumping a core dump file named core. For example:

$ ./a.out
./a.out terminated by signal SIGSEGV (Address boundary error)

We need to load the core dump file in GDB to begin investigating the cause of the error. This is typically done using the command:

$ gdb ./a.out core

GDB loads the program and using the information from the core dump, the program stack and other information is restored to the point where it encountered the error.

A common first step in investigation is print the stack frames at this point:

(gdb) backtrace

Tried with: GDB 7.7.1, GCC 5.1.0 and Ubuntu 14.04

Core file not being overwritten error

Problem

I was dealing with a program that was segfaulting. It was writing a core file on segmentation fault. However, I noticed that if a core file already existed in the directory, the program would not overwrite it on segfault. Only if the existing core file was deleted, then a new one would be created!

Solution

This is due to a well known bug of Apport, the crash reporting service in Ubuntu. It can be disabled as described here. Once disabled, I found that the core file was being overwritten correctly on a new segmentation fault.

Tried with: Apport 2.14.1 and Ubuntu 14.04

How to enable core dump in Fish

When a program crashes due to segmentation fault, then its core dump file is useful to debug the problem. However, when you execute such a program under Fish, it does not create a core file. This is because core dump is disabled by default in Fish. You can check this by using the ulimit command:

$ ulimit -c
0

The output of 0 means that core dump has been disabled.

To enable it you can pass the maximum size of the core file which Fish should allow to be dumped (in KB). If you want to dump core no matter how large the file:

$ ulimit -c unlimited

If you want this setting to be enabled by default, add the above line to your ~/.config/fish/functions/config.fish file. More information on the ulimit command can be found here.

Tried with: Fish 2.0.0 and Ubuntu 14.04