How to set LMDB permissions in Caffe

Caffe can create, read and write images from an LMDB database. This is much faster than using the files from filesystem during training. However, I noticed that by default Caffe would not set write permissions for my group for the lmdb directory.

A quick investigation into src/caffe/util/db_lmdb.cpp showed that this was because new lmdb directory was being created with 0744 permissions. I changed this to 0774 in the code and recompiled Caffe.

Now comes the mystery: Caffe still created lmdb directories where my group did not have write permission!

Further investigation showed that the default umask in Linux is 0022. This umask does not allow the group write permissions to be set.

Ok, so then I set the umask at the shell to 0002 and tried to create the lmdb again. Found that lmdb directories still did not have group write permissions!

I was creating the lmdb databases from a script that ran the convert_imageset binary. Now the binary was run by using the exec command of the Bash shell. This replaces the currently running process with the convert_imageset binary. However, unlike a fork, the new process is not passed the umask of the parent process by the OS! This was the reason for the problem.

Once I knew this, the solution was easy. I modified the tools/convert_imageset.cpp code and added this code:

#include <sys/types.h>
#include <sys/stat.h>

// In the main function ...
umask(0002);
// ....

This solved the problem perfectly! 🙂

Advertisements

How to install cuDNN

cuDNN provides primitives for deep learning networks that have been accelerated for GPUs by NVIDIA.

  • To download cuDNN head over to the cuDNN page here. cuDNN is not directly available for download. NVIDIA requires you to create a login. After that it presents cuDNN downloads in different formats (.tgz or .deb).

  • I prefer to install from .tgz since it gives more control. Unzip the file and it will create a cuda directory which has the required include and lib directories.

  • I like to rename this directory and keep it at /usr/local:

$ mv cuda cudnn
$ mv cudnn /usr/local
  • Remember to add the path to the cuDNN libraries to your LD_LIBRARY_PATH. For my case, that would be /usr/local/cudnn/lib64

  • For CMake in Caffe to automatically find cuDNN while building, export an environment variable named CUDNN_DIR pointing to the directory. For me, that directory would be /usr/local/cudnn

That is it! Caffe should be able to find and link with cuDNN now.

Tried with: cuDNN 6.0 and Ubuntu 16.04

How to build Caffe

Building Caffe from source is easy. There are two options provided: using a Makefile or using CMake.

Required packages

  • Download or clone the source code of Caffe from here.
  • Not surprisingly, you will need a GPU, a graphics driver that can work with that GPU and an installation of CUDA.
  • Other than that, you will need to install these packages:
$ sudo apt install libboost-all-dev libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev
  • For BLAS on CPU, the best performance comes from installing Intel’s MKL libraries. If you do not have that, you can install either ATLAS or OpenBLAS:
$ sudo apt install libatlas-base-dev
$ sudo apt install libopenblas-dev
  • To build the Python interface to Caffe, which is highly recommended, make sure these packages are installed:
$ sudo apt install libboost-python-dev python-skimage python-protobuf
  • If you do not have one or more of the above Python-related packages, you might see these errors:
The dependency target "pycaffe" of target "pytest" does not exist.

ImportError: No module named skimage.io

ImportError: No module named google.protobuf.internal

Build using CMake

Just follow the usual steps for CMake:

$ mkdir build
$ cd build
$ cmake ..
$ make

To build and run the 2000+ unittests, which can take quite a while to finish:

$ make runtest

Finally to build the Python interface to Caffe:

$ make pycaffe
$ make install

By default, the install directory will be a subdirectory inside the build directory. Add this build/install/python path to PYTHONPATH environment variable before you import caffe in Python.

Build using Make

On one system, I found that using Makefile was easier, since CMake was erroneously complaining about -lpthreads (which is a Clang library, not a GCC library).
Copy the Makefile config file, open it in an editor and check whether all the paths are set correctly:

$ cp Makefile.config.example Makefile.config
$ vim Makefile.config
$ make

Build MatCaffe

Building support for MATLAB to use Caffe takes a few more steps:

  • In Makefile.config, set the MATLAB_DIR to the path that contains bin/mex. On my computer, this was the path /usr/local/MATLAB/R2014a.

  • Compiling the MATLAB wrapper gave an error about C++11 on my Ubuntu 14.04 due to the older GCC compilers. So, I had to add -std=c++11 to the CXXFLAGS variable in Makefile as shown here.

  • Finally, compiling MatCaffe is easy:

$ make all matcaffe

Could not create logging file error

Problem

I tried to train a model using Caffe on a remote computer. The training proceeded without any problems, but this error would show up in the console output occasionally:

Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170221-111719.33617!I0221 11:17:19.399444 33617 solver.cpp:243] Iteration 86860, loss = 3.38734

Solution

This error turned out to be from GLog, the Google logging module. Caffe uses GLog for its logging. On this particular remote computer, the GLOG_log_dir environment variable had been set, but to a non-existent path! This was the reason GLog was complaining. Once I removed this environment variable and restarted the training, I no longer saw the error.

Tried with: Ubuntu 14.04

Error building Caffe with Python 3 support

Caffe can be built with support for Python 2.x. This allows you to invoke Caffe easily from Python code. However, I wanted to call Caffe from Python 3.x code.

  • I built Boost with Python 3.x support. I could see that libboost_python3 library files were generated.

  • I added this to the normal CMake command that I use to build Caffe: -Dpython_version=3

Sadly, this popped up errors of this type:

libboost_python.so: undefined reference to `PyClass_Type'

This type of error indicates that the Python 2.x Boost library was being used to compile with Python 3.x libraries.

Netscope

Netscope provides a beautiful visualization in your browser of deep neural network architectures like CNN.

  • It requires input to be in the Caffe ProtoBuf format.
  • It is a good alternative to using GraphViz as described here. Compared to GraphViz, Netscope uses better default colors, the graph is more orderly and most importantly, hovering over the nodes provides all the details.

Try it out!

How to visualize Caffe Net using GraphViz

The network architecture of Convolutional Neural Networks (CNN) can be heavily layered and complex. Viewing the network visually is a great way to get a sense of its architecture. Since the network is a graph, it is easy to visualize this using GraphViz.

Caffe requires its Net to be in the Google ProtoBuf format. It also provides a draw_net.py script that can be used to output the graph to all the formats supported by GraphViz.

  • From the Caffe root directory, you can export a .prototxt model file as a graph to a PNG image file:
$ python/draw_net.py foo.prototxt foo.png

Possible output formats include PNG, PDF, DOT and others supported by GraphViz.

  • By default, the net layers are drawn from left-to-right. I prefer to visualize a CNN in top-to-bottom fashion:
$ python/draw_net.py --rankdir TB foo.prototxt foo.png
  • I prefer to interact with the graph visualization, which is a bit difficult with an image file. So, I prefer to export to DOT format file and play with it using XDot:
$ python/draw_net.py foo.prototxt foo.dot
$ xdot foo.dot

Tried with: Ubuntu 14.04