Read and write same file using sponge

A common operation I end up doing is reading a text file, processing it using Unix tools and writing back the result to the same filename. However, if you do this you will find that you end up with an empty file! This is because the file is first opened for writing, thus clearing its contents.

sponge is a tiny tool created specially for this common operation. It can be installed easily:

$ sudo apt install moreutils

Here is an example that is wrong and ends up creating an empty file:

$ cat foobar.txt | uniq > foobar.txt

To fix this, add sponge to soak up all the output first and only write to the file at the end:

$ cat foobar.txt | uniq | sponge foobar.txt

Tried with: moreutils 0.57 and Ubuntu 16.04

Advertisements

Process JSON at the shell using jq

jq is an awesome tool to play with JSON data at the commandline.

  • Installing it is easy:
$ sudo apt install jq
  • To format a JSON file so that it looks pretty:
$ cat ugly.json | jq '.'

I have found this extremely useful to format huge JSON files. jq is much faster at this job compared to the json.tool in Python.

  • You can access and print a specific section of the JSON just like in Python. For example:
$ cat foobar.json | jq '.["records"][6]["name"]'

Note the use of single quotes to encapsulate the expression and use of double quotes inside to specify keys in dictionaries.

Tried with: jq 1.5 and Ubuntu 16.04

Compare files at shell using comm

comm

comm is a useful Linux tool to use at the shell. It takes two files with sorted lines as input and displays which lines are unique to each file and which lines are common (intersection) to both. For example, this might be useful when comparing which files are common among two directories and so on. This tool is a part of coreutils package, so it should be available everywhere.

  • To compare two files:
$ comm first.txt second.txt

You will see 3 columns in the output corresponding to lines unique to first file, lines unique to second file and lines common to both.

  • To suppress columns 1 and 2, thus display only columns common to both files:
$ comm -1 -2 first.txt second.txt
  • There are a few other options to this tool which can be read about in the manpage: man comm

Tried with: Ubuntu 16.04

Python JSON dump misses last newline

Problem

The dump method from the Python json package can be used to write a suitable Python object, usually a dictionary or list, to a JSON file. However, I discovered that Unix shell programs have problems working with such a JSON file. This turned out to be because this dump method does not end the last line with a newline character! According to the POSIX definition of a line in a text file, it needs to end with a newline character. (See here).

Solution

I replaced this:

json.dump(json_data, open("foobar.json", "w"), indent=4)

with this:

with open("foobar.json", "w") as json_file:
    json_text = json.dumps(json_data, indent=4)
    json_file.write("{}\n".format(json_text))  # Add newline cause Py does not

How to build Caffe

Building Caffe from source is easy. There are two options provided: using a Makefile or using CMake. I found that using Makefile was easier, since CMake was erroneously complaining about -lpthreads (which is a Clang library, not a GCC library).

  • Download or clone the source code of Caffe from here.

  • Copy the Makefile config file, open it in an editor and check whether all the paths are set correctly:

$ cp Makefile.config.example Makefile.config
$ vim Makefile.config
  • To build Caffe:
$ make all

MatCaffe

Building support for MATLAB to use Caffe takes a few more steps:

  • In Makefile.config, set the MATLAB_DIR to the path that contains bin/mex. On my computer, this was the path /usr/local/MATLAB/R2014a.

  • Compiling the MATLAB wrapper gave an error about C++11 on my Ubuntu 14.04 due to the older GCC compilers. So, I had to add -std=c++11 to the CXXFLAGS variable in Makefile as shown here.

  • Finally, compiling MatCaffe is easy:

$ make all matcaffe

Could not create logging file error

Problem

I tried to train a model using Caffe on a remote computer. The training proceeded without any problems, but this error would show up in the console output occasionally:

Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170221-111719.33617!I0221 11:17:19.399444 33617 solver.cpp:243] Iteration 86860, loss = 3.38734

Solution

This error turned out to be from GLog, the Google logging module. Caffe uses GLog for its logging. On this particular remote computer, the GLOG_log_dir environment variable had been set, but to a non-existent path! This was the reason GLog was complaining. Once I removed this environment variable and restarted the training, I no longer saw the error.

Tried with: Ubuntu 14.04

ImageMagick display

ImageMagick display

The display program of ImageMagick is useful when you are at the shell and need to quickly view one or a few images. It can also be used to apply many of the image processing operations available in ImageMagick on an image and view it. This viewer is extremely light and great for checking some images quickly on a remote computer over SSH.

  • If you do not have ImageMagick, installing it is easy:
$ sudo apt install imagemagick
  • To view a specific image:
$ display foo.jpg
  • To view all the JPEG files in current directory:
$ display *.jpg
  • To display all image files in current directory:
$ display *
  • If you know that you will not be changing the images, but just viewing them, then open them in immutable mode:
$ display -immutable *.jpg
  • Press F1 to view help about the mouse operations and keyboard operations available in this program.

  • Some of the keyboard shortcuts I frequently use:

Space: Next image
Backspace: Previous image
Esc: Quit
/: Rotate by 90 degrees
<: Zoom out
>: Zoom in

Tried with: ImageMagick 6.8.9.9 and Ubuntu 16.04

How to install Ubuntu on computer with UEFI

  • Unified Extensible Firmware Interface (UEFI) is a modern replacement over the BIOS for booting up operating systems. In most BIOS settings, you can choose whether you want UEFI enabled or disabled.

  • Ubuntu works best with BIOS, not UEFI. So, if you are only installing Ubuntu on a computer or you have Windows 7 or older on it then you can disable UEFI in the BIOS settings and proceed to install Ubuntu with a / partition and a swap partition.

  • However, if you already have Windows 8 or 10 installed on the computer, it will be using UEFI to boot and so UEFI cannot be disabled. Also, if you intend to install and dual boot with Windows in the future, it may be a good idea to keep UEFI enabled.

  • If you already have Windows 8 or 10 installed then you will notice an extra 100MB partition named EFI while installing Ubuntu. Do not touch it and proceed with creating a / and swap partition. Ubuntu will automatically install GRUB to use that EFI partition for booting.

  • If you do not have Windows, but want to keep UEFI enabled, then you will need to create a 100-200MB partition and pick its type as boot EFI while installing Ubuntu. The rest of the partitions for Ubuntu are the same. Just remember to create an EFI partition.

  • If you forgot to create an EFI partition on a UEFI enabled computer and installed Ubuntu, then the installer will fail at the end with this error:

The `grub-efi-amd64-signed` package failed to install into / target/.
Without the GRUB boot loader, the installed system will not boot.

Tried with: Ubuntu 16.04

How to make Intel Wireless-AC 3165 work in Ubuntu

Problem

I installed Ubuntu 16.04 on a new notebook that had Intel Wireless-AC 3165 hardware to provide wifi and Bluetooth connectivity. This hardware was working without any problems in Windows 10, without requiring any extra software installation. However, the wireless and Bluetooth features did not work in Ubuntu.

Solution

There are many solutions offered on the web to enable wireless connectivity. Copying the ucode files provided by Intel here to /lib/firmware definitely did not work.

I was using Ubuntu 16.04, which was using the Linux 4.4.x kernel. One of the suggestions online is that newer kernels have better support for Intel wireless hardware.

Ubuntu developers provide installers of all newer Linux kernels. So I decided to try my luck with the latest 4.9.x series and installed it as described here. I restarted the notebook and had wireless and Bluetooth working like magic!

How to update Linux kernel in Ubuntu

Every version of Ubuntu LTS sticks to a particular version of the Linux kernel. For example, Ubuntu 16.04 sticks to the 4.4.x series of kernels. Over the months and years you use and update this version of Ubuntu, only newer minor versions of the kernel are updated to maintain stability.

However, there might be reasons you might want to upgrade to a major new Linux kernel version. For example, to get support for newer hardware and firmware. For such options, Ubuntu developers maintain a series of the Linux mainline kernel which can be downloaded and installed.

  • Go to http://kernel.ubuntu.com/~kernel-ppa/mainline/ and decide which version of kernel you want. For example, I decided to upgrade from 4.4.x to 4.9.x.

  • You need to download 3 deb files for a full kernel installation. These will be named in this format: linux-headers-XXX_amd64.deb, linux-headers-XXX_all.deb and linux-image-XXX-generic_XXX_amd64.deb.

  • Install them:

$ sudo dpkg -i *.deb
  • Restart the computer and check if you are using the new kernel:
$ uname -r
  • Do not delete the kernel provided by Ubuntu even if apt keeps reminding you that you do not need it! If something goes wrong with the new kernel, you might want to keep the older one, so you can boot using that in the GRUB boot screen.