Caffe can create, read and write images from an LMDB database. This is much faster than using the files from filesystem during training. However, I noticed that by default Caffe would not set write permissions for my group for the lmdb directory.
A quick investigation into src/caffe/util/db_lmdb.cpp showed that this was because new lmdb directory was being created with 0744 permissions. I changed this to 0774 in the code and recompiled Caffe.
Now comes the mystery: Caffe still created lmdb directories where my group did not have write permission!
Further investigation showed that the default umask in Linux is 0022. This umask does not allow the group write permissions to be set.
Ok, so then I set the umask at the shell to 0002 and tried to create the lmdb again. Found that lmdb directories still did not have group write permissions!
I was creating the lmdb databases from a script that ran the convert_imageset binary. Now the binary was run by using the exec command of the Bash shell. This replaces the currently running process with the convert_imageset binary. However, unlike a fork, the new process is not passed the umask of the parent process by the OS! This was the reason for the problem.
Once I knew this, the solution was easy. I modified the tools/convert_imageset.cpp code and added this code:
#include <sys/types.h>
#include <sys/stat.h>
// In the main function ...
umask(0002);
// ....This solved the problem perfectly! :)