Caffe can create, read and write images from an LMDB database. This is much faster than using the files from filesystem during training. However, I noticed that by default Caffe would not set write permissions for my group for the lmdb directory.
A quick investigation into
src/caffe/util/db_lmdb.cpp showed that this was because new lmdb directory was being created with
0744 permissions. I changed this to
0774 in the code and recompiled Caffe.
Now comes the mystery: Caffe still created lmdb directories where my group did not have write permission!
Further investigation showed that the default umask in Linux is
0022. This umask does not allow the group write permissions to be set.
Ok, so then I set the umask at the shell to
0002 and tried to create the lmdb again. Found that lmdb directories still did not have group write permissions!
I was creating the lmdb databases from a script that ran the
convert_imageset binary. Now the binary was run by using the
exec command of the Bash shell. This replaces the currently running process with the
convert_imageset binary. However, unlike a fork, the new process is not passed the umask of the parent process by the OS! This was the reason for the problem.
Once I knew this, the solution was easy. I modified the
tools/convert_imageset.cpp code and added this code:
// In the main function ...
This solved the problem perfectly! 🙂
cuDNN provides primitives for deep learning networks that have been accelerated for GPUs by NVIDIA.
- To download cuDNN head over to the cuDNN page here. cuDNN is not directly available for download. NVIDIA requires you to create a login. After that it presents cuDNN downloads in different formats (.tgz or .deb).
I prefer to install from .tgz since it gives more control. Unzip the file and it will create a
cuda directory which has the required include and lib directories.
I like to rename this directory and keep it at
$ mv cuda cudnn
$ mv cudnn /usr/local
Remember to add the path to the cuDNN libraries to your
LD_LIBRARY_PATH. For my case, that would be
For CMake in Caffe to automatically find cuDNN while building, export an environment variable named
CUDNN_DIR pointing to the directory. For me, that directory would be
That is it! Caffe should be able to find and link with cuDNN now.
Tried with: cuDNN 6.0 and Ubuntu 16.04
Building Caffe from source is easy. There are two options provided: using a Makefile or using CMake.
- Download or clone the source code of Caffe from here.
- Not surprisingly, you will need a GPU, a graphics driver that can work with that GPU and an installation of CUDA.
- Other than that, you will need to install these packages:
$ sudo apt install libboost-all-dev libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev
- For BLAS on CPU, the best performance comes from installing Intel’s MKL libraries. If you do not have that, you can install either ATLAS or OpenBLAS:
$ sudo apt install libatlas-base-dev
$ sudo apt install libopenblas-dev
- To build the Python interface to Caffe, which is highly recommended, make sure these packages are installed:
$ sudo apt install libboost-python-dev python-skimage python-protobuf
- If you do not have one or more of the above Python-related packages, you might see these errors:
The dependency target "pycaffe" of target "pytest" does not exist.
ImportError: No module named skimage.io
ImportError: No module named google.protobuf.internal
Build using CMake
Just follow the usual steps for CMake:
$ mkdir build
$ cd build
$ cmake ..
To build and run the 2000+ unittests, which can take quite a while to finish:
$ make runtest
Finally to build the Python interface to Caffe:
$ make pycaffe
$ make install
By default, the install directory will be a subdirectory inside the
build directory. Add this
build/install/python path to
PYTHONPATH environment variable before you import caffe in Python.
Build using Make
On one system, I found that using Makefile was easier, since CMake was erroneously complaining about
-lpthreads (which is a Clang library, not a GCC library).
Copy the Makefile config file, open it in an editor and check whether all the paths are set correctly:
$ cp Makefile.config.example Makefile.config
$ vim Makefile.config
Building support for MATLAB to use Caffe takes a few more steps:
In Makefile.config, set the
MATLAB_DIR to the path that contains
bin/mex. On my computer, this was the path
Compiling the MATLAB wrapper gave an error about C++11 on my Ubuntu 14.04 due to the older GCC compilers. So, I had to add
-std=c++11 to the
CXXFLAGS variable in
Makefile as shown here.
Finally, compiling MatCaffe is easy:
$ make all matcaffe
I tried to train a model using Caffe on a remote computer. The training proceeded without any problems, but this error would show up in the console output occasionally:
Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170221-111719.33617!I0221 11:17:19.399444 33617 solver.cpp:243] Iteration 86860, loss = 3.38734
This error turned out to be from GLog, the Google logging module. Caffe uses GLog for its logging. On this particular remote computer, the
GLOG_log_dir environment variable had been set, but to a non-existent path! This was the reason GLog was complaining. Once I removed this environment variable and restarted the training, I no longer saw the error.
Tried with: Ubuntu 14.04
Caffe can be built with support for Python 2.x. This allows you to invoke Caffe easily from Python code. However, I wanted to call Caffe from Python 3.x code.
Sadly, this popped up errors of this type:
libboost_python.so: undefined reference to `PyClass_Type'
This type of error indicates that the Python 2.x Boost library was being used to compile with Python 3.x libraries.
Netscope provides a beautiful visualization in your browser of deep neural network architectures like CNN.
- It requires input to be in the Caffe ProtoBuf format.
- It is a good alternative to using GraphViz as described here. Compared to GraphViz, Netscope uses better default colors, the graph is more orderly and most importantly, hovering over the nodes provides all the details.
Try it out!
The network architecture of Convolutional Neural Networks (CNN) can be heavily layered and complex. Viewing the network visually is a great way to get a sense of its architecture. Since the network is a graph, it is easy to visualize this using GraphViz.
Caffe requires its Net to be in the Google ProtoBuf format. It also provides a
draw_net.py script that can be used to output the graph to all the formats supported by GraphViz.
- From the Caffe root directory, you can export a
.prototxt model file as a graph to a PNG image file:
$ python/draw_net.py foo.prototxt foo.png
Possible output formats include PNG, PDF, DOT and others supported by GraphViz.
- By default, the net layers are drawn from left-to-right. I prefer to visualize a CNN in top-to-bottom fashion:
$ python/draw_net.py --rankdir TB foo.prototxt foo.png
- I prefer to interact with the graph visualization, which is a bit difficult with an image file. So, I prefer to export to DOT format file and play with it using XDot:
$ python/draw_net.py foo.prototxt foo.dot
$ xdot foo.dot
Tried with: Ubuntu 14.04