Aligned memory allocation

In some scenarios, you want to get memory that is aligned at an address that is a certain power of 2. Certain CPU architectures and certain operations require (or are faster) if their operands are located at an address that is a multiple of a certain power-of-2 number. For these reasons, you might see that many multi-platform libraries use an aligned memory allocator instead of malloc in their code. For example, OpenCV uses methods named fastMalloc and fastFree inside its code that do this type of allocation and freeing.

Most of these methods work like this:

  • They internally get memory from malloc. However, if you requested for N bytes, the wrapper will request for N+P+A bytes from malloc. Here, P is the size of a pointer on that CPU architecture and A is the alignment required, expressed in power-of-2 number of bytes. For example, if I request for 100 bytes on a 64-bit CPU and require the memory to be aligned to a multiple of 32, then the wrapper will request for 140 bytes.

  • After getting the memory from malloc, it aligns the pointer forward so that (1) the pointer is at an address that is aligned as per requirement and (2) there is space behind the pointer to store a memory address.

  • Then we sneak and store the address actually returned by malloc behind the pointer address and return the pointer to the user.

  • The user has to use our free wrapper to free this pointer. When she does that we sneak back to reveal the actual address returned by malloc and free using that.

Here is some example code that illustrates aligned memory allocation:

Advertisements

isinstance and issubclass in Python

isinstance

Use this built-in function to find out if a given object is an instance of a certain class or any of its subclasses. You can even pass a tuple of classes to be checked for the object.

The only gotcha you will discover is this: in Python a bool object is an instance of int! Yes, not kidding!

issubclass

This built-in function is similar to isinstance, but to check if a type is an instance of a class or any of its subclasses.

Again, the only gotcha is that bool type is subclass of int type.

How to convert Python dict to class object with fields

A Python dict is extremely versatile. However, there might be situations where you want to access the dict as if it were a class object and the keys were its fields. This can be easily done by using a namedtuple. Just give it a name and the use the keys to populate its named fields. Set the values for those fields by passing the values from the dict. It all boils down to a single line.

This example code demonstrates the above:

pgrep and pkill

pgrep and pkill are two useful commands that go well together. You can list processes using ps and kill them using kill. However, I find it easier to use pgrep and pkill when I want to find and kill a process.

pgrep

  • To list all the PIDs of a user:
$ pgrep --uid joe
$ pgrep -u joe
  • To list the PID and the corresponding process name for a user:
$ pgrep --uid joe --list-name
$ pgrep -u joe -l
  • To list the PID and the corresponding full command line for a user:
$ pgrep --uid joe --list-full
$ pgrep -u joe -a

This is extremely useful because to find Python scripts or commandline arguments to a program that is running as a process.

  • To list the PID whose process name matches input pattern:
$ pgrep foobar
  • To list the PID and process names that match input pattern:
$ pgrep -l foobar
  • To list the PID and command line of processes that match input pattern:
$ pgrep -a foobar

pkill

pkill is used to send a signal to or kill processes by using a pattern that matches a process name or its command line. pkill takes many arguments that are similar to pgrep.

  • To kill all processes of current user that matches input pattern:
$ pkill foobar
  • To kill all processes of current user that matches input pattern in its command line:
$ pkill -f foobar

Tried with: Ubuntu 16.04

Read and write same file using sponge

A common operation I end up doing is reading a text file, processing it using Unix tools and writing back the result to the same filename. However, if you do this you will find that you end up with an empty file! This is because the file is first opened for writing, thus clearing its contents.

sponge is a tiny tool created specially for this common operation. It can be installed easily:

$ sudo apt install moreutils

Here is an example that is wrong and ends up creating an empty file:

$ cat foobar.txt | uniq > foobar.txt

To fix this, add sponge to soak up all the output first and only write to the file at the end:

$ cat foobar.txt | uniq | sponge foobar.txt

Tried with: moreutils 0.57 and Ubuntu 16.04

Process JSON at the shell using jq

jq is an awesome tool to play with JSON data at the commandline.

  • Installing it is easy:
$ sudo apt install jq
  • To format a JSON file so that it looks pretty:
$ cat ugly.json | jq '.'

I have found this extremely useful to format huge JSON files. jq is much faster at this job compared to the json.tool in Python.

  • You can access and print a specific section of the JSON just like in Python. For example:
$ cat foobar.json | jq '.["records"][6]["name"]'

Note the use of single quotes to encapsulate the expression and use of double quotes inside to specify keys in dictionaries.

Tried with: jq 1.5 and Ubuntu 16.04

Compare files at shell using comm

comm

comm is a useful Linux tool to use at the shell. It takes two files with sorted lines as input and displays which lines are unique to each file and which lines are common (intersection) to both. For example, this might be useful when comparing which files are common among two directories and so on. This tool is a part of coreutils package, so it should be available everywhere.

  • To compare two files:
$ comm first.txt second.txt

You will see 3 columns in the output corresponding to lines unique to first file, lines unique to second file and lines common to both.

  • To suppress columns 1 and 2, thus display only columns common to both files:
$ comm -1 -2 first.txt second.txt
  • There are a few other options to this tool which can be read about in the manpage: man comm

Tried with: Ubuntu 16.04

Python JSON dump misses last newline

Problem

The dump method from the Python json package can be used to write a suitable Python object, usually a dictionary or list, to a JSON file. However, I discovered that Unix shell programs have problems working with such a JSON file. This turned out to be because this dump method does not end the last line with a newline character! According to the POSIX definition of a line in a text file, it needs to end with a newline character. (See here).

Solution

I replaced this:

json.dump(json_data, open("foobar.json", "w"), indent=4)

with this:

with open("foobar.json", "w") as json_file:
    json_text = json.dumps(json_data, indent=4)
    json_file.write("{}\n".format(json_text))  # Add newline cause Py does not

How to build Caffe

Building Caffe from source is easy. There are two options provided: using a Makefile or using CMake.

Required packages

  • Download or clone the source code of Caffe from here.
  • Not surprisingly, you will need a GPU, a graphics driver that can work with that GPU and an installation of CUDA.
  • Other than that, you will need to install these packages:
$ sudo apt install libboost-all-dev libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev
  • For BLAS on CPU, the best performance comes from installing Intel’s MKL libraries. If you do not have that, you can install either ATLAS or OpenBLAS:
$ sudo apt install libatlas-base-dev
$ sudo apt install libopenblas-dev
  • To build the Python interface to Caffe, which is highly recommended, make sure these packages are installed:
$ sudo apt install libboost-python-dev python-skimage python-protobuf
  • If you do not have one or more of the above Python-related packages, you might see these errors:
The dependency target "pycaffe" of target "pytest" does not exist.

ImportError: No module named skimage.io

ImportError: No module named google.protobuf.internal

Build using CMake

Just follow the usual steps for CMake:

$ mkdir build
$ cd build
$ cmake ..
$ make

To build and run the 2000+ unittests, which can take quite a while to finish:

$ make runtest

Finally to build the Python interface to Caffe:

$ make pycaffe
$ make install

By default, the install directory will be a subdirectory inside the build directory. Add this build/install/python path to PYTHONPATH environment variable before you import caffe in Python.

Build using Make

On one system, I found that using Makefile was easier, since CMake was erroneously complaining about -lpthreads (which is a Clang library, not a GCC library).
Copy the Makefile config file, open it in an editor and check whether all the paths are set correctly:

$ cp Makefile.config.example Makefile.config
$ vim Makefile.config
$ make

Build MatCaffe

Building support for MATLAB to use Caffe takes a few more steps:

  • In Makefile.config, set the MATLAB_DIR to the path that contains bin/mex. On my computer, this was the path /usr/local/MATLAB/R2014a.

  • Compiling the MATLAB wrapper gave an error about C++11 on my Ubuntu 14.04 due to the older GCC compilers. So, I had to add -std=c++11 to the CXXFLAGS variable in Makefile as shown here.

  • Finally, compiling MatCaffe is easy:

$ make all matcaffe

Could not create logging file error

Problem

I tried to train a model using Caffe on a remote computer. The training proceeded without any problems, but this error would show up in the console output occasionally:

Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170221-111719.33617!I0221 11:17:19.399444 33617 solver.cpp:243] Iteration 86860, loss = 3.38734

Solution

This error turned out to be from GLog, the Google logging module. Caffe uses GLog for its logging. On this particular remote computer, the GLOG_log_dir environment variable had been set, but to a non-existent path! This was the reason GLog was complaining. Once I removed this environment variable and restarted the training, I no longer saw the error.

Tried with: Ubuntu 14.04