How to debug Caffe

Caffe is written in C++ while its GPU accelerated portions are written in CUDA. It also ships with PyCaffe, a Python wrapper to the C++ code.

To debug Python code of PyCaffe

You might have written a Python script to train or load and use models for inference. The Python code of Caffe can be debugged as usual using the PDB Python debugger as described here.

To debug C++ code from Caffe binary

We can use GDB to debug the C++ code in Caffe. First, remember to build Caffe with debugging information. This can be done as described here, but remember to indicate Debug mode to CMake, like this:

$ cmake -DCMAKE_BUILD_TYPE=Debug ..

If you are using binary Caffe, then debugging the C++ code is straightforward using GDB:

$ gdb --args ./caffe --your-usual-caffe-arguments-go-here

You can set breakpoints in any Caffe C++ code and debug using GDB as usual. For more info see my GDB cheatsheet.

To debug C++ code from PyCaffe

If you are using PyCaffe and need to debug the C++ parts of Caffe code that is still possible using GDB! Remember to first build Caffe in Debug mode as shown above. Note that after make install, you might have to rename the file to to be able to import caffe in your Python script.

To be able to debug the C++ code, we invoke GDB and pass it the python interpreter and its arguments, which is our Python script that calls PyCaffe:

$ gdb --args python --some_input_arguments

GDB first begins in the Python interpreter binary. Python will later load all the required shared libraries including and more importantly which is the compiled version of the Caffe C++ code. The problem is that these libraries are not yet loaded at the beginning. What we can do is set one or more breakpoints at the places we want right away. GDB will complain that these breakpoint locations are as yet not known to it. That is okay, it will turn them into pending breakpoints that will be enabled when the shared library having that location is loaded.

For example, to set up a breakpoint right away in convolution layer code:

(gdb) b base_conv_layer.cpp:15

After you have set your breakpoints, press c to continue and GDB will load the shared libraries and stop when your breakpoint locations are hit. After this point it is debugging as usual in GDB.

If you are adventurous, you can even connect GDB to the Python process that is running PyCaffe/Caffe by using its PID. For example, after I find out that my Python script is running as PID 589:

$ gdb -p 589

You can set breakpoints in Caffe C++ code and GDB will stop at those locations.

I hope you have fun stepping through and exploring Caffe code! 🙂

Tried with: GDB 7.7.1 and Ubuntu 14.04


Error using TensorRT engine for inference


I used TensorRT to convert a Caffe model into an engine (plan) file on one computer. When I tried to load this engine (plan) file on another computer and use it for inference using TensorRT, I got this error:

customWinogradConvActLayer.cpp:195: virtual void nvinfer1::cudnn::WinogradConvActLayer::allocateResources(const nvinfer1::cudnn::CommonContext&): Assertion 'configIsValid(context)' failed.


It turns out that the first computer had a NVIDIA 1080 Ti GPU and the engine had been created for it. The second computer had a NVIDIA K80 GPU. Though, TensorRT documentation is vague about this, it seems like an engine created on a specific GPU can only be used for inference on the same model of GPU!

When I created a plan file on the K80 computer, inference worked fine.

Tried with: TensorRT 2.1, cuDNN 6.0 and CUDA 8.0

How to set LMDB permissions in Caffe

Caffe can create, read and write images from an LMDB database. This is much faster than using the files from filesystem during training. However, I noticed that by default Caffe would not set write permissions for my group for the lmdb directory.

A quick investigation into src/caffe/util/db_lmdb.cpp showed that this was because new lmdb directory was being created with 0744 permissions. I changed this to 0774 in the code and recompiled Caffe.

Now comes the mystery: Caffe still created lmdb directories where my group did not have write permission!

Further investigation showed that the default umask in Linux is 0022. This umask does not allow the group write permissions to be set.

Ok, so then I set the umask at the shell to 0002 and tried to create the lmdb again. Found that lmdb directories still did not have group write permissions!

I was creating the lmdb databases from a script that ran the convert_imageset binary. Now the binary was run by using the exec command of the Bash shell. This replaces the currently running process with the convert_imageset binary. However, unlike a fork, the new process is not passed the umask of the parent process by the OS! This was the reason for the problem.

Once I knew this, the solution was easy. I modified the tools/convert_imageset.cpp code and added this code:

#include <sys/types.h>
#include <sys/stat.h>

// In the main function ...
// ....

This solved the problem perfectly! 🙂

How to install cuDNN

cuDNN provides primitives for deep learning networks that have been accelerated for GPUs by NVIDIA.

  • To download cuDNN head over to the cuDNN page here. cuDNN is not directly available for download. NVIDIA requires you to create a login. After that it presents cuDNN downloads in different formats (.tgz or .deb).

  • I prefer to install from .tgz since it gives more control. Unzip the file and it will create a cuda directory which has the required include and lib directories.

  • I like to rename this directory and keep it at /usr/local:

$ mv cuda cudnn
$ mv cudnn /usr/local
  • Remember to add the path to the cuDNN libraries to your LD_LIBRARY_PATH. For my case, that would be /usr/local/cudnn/lib64

  • For CMake in Caffe to automatically find cuDNN while building, export an environment variable named CUDNN_DIR pointing to the directory. For me, that directory would be /usr/local/cudnn

That is it! Caffe should be able to find and link with cuDNN now.

Tried with: cuDNN 6.0 and Ubuntu 16.04

How to build Caffe

Building Caffe from source is easy. There are two options provided: using a Makefile or using CMake.

Required packages

  • Download or clone the source code of Caffe from here.
  • Not surprisingly, you will need a GPU, a graphics driver that can work with that GPU and an installation of CUDA.
  • Other than that, you will need to install these packages:
$ sudo apt install libboost-all-dev libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev
  • For BLAS on CPU, the best performance comes from installing Intel’s MKL libraries. If you do not have that, you can install either ATLAS or OpenBLAS:
$ sudo apt install libatlas-base-dev
$ sudo apt install libopenblas-dev
  • To build the Python interface to Caffe, which is highly recommended, make sure these packages are installed:
$ sudo apt install libboost-python-dev python-skimage python-protobuf
  • If you do not have one or more of the above Python-related packages, you might see these errors:
The dependency target "pycaffe" of target "pytest" does not exist.

ImportError: No module named

ImportError: No module named google.protobuf.internal

Build using CMake

Just follow the usual steps for CMake:

$ mkdir build
$ cd build
$ cmake ..
$ make

To build and run the 2000+ unittests, which can take quite a while to finish:

$ make runtest

Finally to build the Python interface to Caffe:

$ make pycaffe
$ make install

By default, the install directory will be a subdirectory inside the build directory. Add this build/install/python path to PYTHONPATH environment variable before you import caffe in Python.

Build using Make

On one system, I found that using Makefile was easier, since CMake was erroneously complaining about -lpthreads (which is a Clang library, not a GCC library).
Copy the Makefile config file, open it in an editor and check whether all the paths are set correctly:

$ cp Makefile.config.example Makefile.config
$ vim Makefile.config
$ make

Build MatCaffe

Building support for MATLAB to use Caffe takes a few more steps:

  • In Makefile.config, set the MATLAB_DIR to the path that contains bin/mex. On my computer, this was the path /usr/local/MATLAB/R2014a.

  • Compiling the MATLAB wrapper gave an error about C++11 on my Ubuntu 14.04 due to the older GCC compilers. So, I had to add -std=c++11 to the CXXFLAGS variable in Makefile as shown here.

  • Finally, compiling MatCaffe is easy:

$ make all matcaffe

Could not create logging file error


I tried to train a model using Caffe on a remote computer. The training proceeded without any problems, but this error would show up in the console output occasionally:

Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170221-111719.33617!I0221 11:17:19.399444 33617 solver.cpp:243] Iteration 86860, loss = 3.38734


This error turned out to be from GLog, the Google logging module. Caffe uses GLog for its logging. On this particular remote computer, the GLOG_log_dir environment variable had been set, but to a non-existent path! This was the reason GLog was complaining. Once I removed this environment variable and restarted the training, I no longer saw the error.

Tried with: Ubuntu 14.04

Error building Caffe with Python 3 support

Caffe can be built with support for Python 2.x. This allows you to invoke Caffe easily from Python code. However, I wanted to call Caffe from Python 3.x code.

  • I built Boost with Python 3.x support. I could see that libboost_python3 library files were generated.

  • I added this to the normal CMake command that I use to build Caffe: -Dpython_version=3

Sadly, this popped up errors of this type: undefined reference to `PyClass_Type'

This type of error indicates that the Python 2.x Boost library was being used to compile with Python 3.x libraries.


Netscope provides a beautiful visualization in your browser of deep neural network architectures like CNN.

  • It requires input to be in the Caffe ProtoBuf format.
  • It is a good alternative to using GraphViz as described here. Compared to GraphViz, Netscope uses better default colors, the graph is more orderly and most importantly, hovering over the nodes provides all the details.

Try it out!

How to visualize Caffe Net using GraphViz

The network architecture of Convolutional Neural Networks (CNN) can be heavily layered and complex. Viewing the network visually is a great way to get a sense of its architecture. Since the network is a graph, it is easy to visualize this using GraphViz.

Caffe requires its Net to be in the Google ProtoBuf format. It also provides a script that can be used to output the graph to all the formats supported by GraphViz.

  • From the Caffe root directory, you can export a .prototxt model file as a graph to a PNG image file:
$ python/ foo.prototxt foo.png

Possible output formats include PNG, PDF, DOT and others supported by GraphViz.

  • By default, the net layers are drawn from left-to-right. I prefer to visualize a CNN in top-to-bottom fashion:
$ python/ --rankdir TB foo.prototxt foo.png
  • I prefer to interact with the graph visualization, which is a bit difficult with an image file. So, I prefer to export to DOT format file and play with it using XDot:
$ python/ foo.prototxt
$ xdot

Tried with: Ubuntu 14.04