How to debug Caffe

Caffe is written in C++ while its GPU accelerated portions are written in CUDA. It also ships with PyCaffe, a Python wrapper to the C++ code.

To debug Python code of PyCaffe

You might have written a Python script to train or load and use models for inference. The Python code of Caffe can be debugged as usual using the PDB Python debugger as described here.

To debug C++ code from Caffe binary

We can use GDB to debug the C++ code in Caffe. First, remember to build Caffe with debugging information. This can be done as described here, but remember to indicate Debug mode to CMake, like this:

$ cmake -DCMAKE_BUILD_TYPE=Debug ..

If you are using binary Caffe, then debugging the C++ code is straightforward using GDB:

$ gdb --args ./caffe --your-usual-caffe-arguments-go-here

You can set breakpoints in any Caffe C++ code and debug using GDB as usual. For more info see my GDB cheatsheet.

To debug C++ code from PyCaffe

If you are using PyCaffe and need to debug the C++ parts of Caffe code that is still possible using GDB! Remember to first build Caffe in Debug mode as shown above. Note that after make install, you might have to rename the _caffe-d.so file to _caffe.so to be able to import caffe in your Python script.

To be able to debug the C++ code, we invoke GDB and pass it the python interpreter and its arguments, which is our Python script that calls PyCaffe:

$ gdb --args python my_script_calls_pycaffe.py --some_input_arguments

GDB first begins in the Python interpreter binary. Python will later load all the required shared libraries including _caffe.so and more importantly libcaffe-nv.so which is the compiled version of the Caffe C++ code. The problem is that these libraries are not yet loaded at the beginning. What we can do is set one or more breakpoints at the places we want right away. GDB will complain that these breakpoint locations are as yet not known to it. That is okay, it will turn them into pending breakpoints that will be enabled when the shared library having that location is loaded.

For example, to set up a breakpoint right away in convolution layer code:

(gdb) b base_conv_layer.cpp:15

After you have set your breakpoints, press c to continue and GDB will load the shared libraries and stop when your breakpoint locations are hit. After this point it is debugging as usual in GDB.

If you are adventurous, you can even connect GDB to the Python process that is running PyCaffe/Caffe by using its PID. For example, after I find out that my Python script is running as PID 589:

$ gdb -p 589

You can set breakpoints in Caffe C++ code and GDB will stop at those locations.

I hope you have fun stepping through and exploring Caffe code! 😊

Tried with: GDB 7.7.1 and Ubuntu 14.04