Python JSON dump misses last newline

Problem

The dump method from the Python json package can be used to write a suitable Python object, usually a dictionary or list, to a JSON file. However, I discovered that Unix shell programs have problems working with such a JSON file. This turned out to be because this dump method does not end the last line with a newline character! According to the POSIX definition of a line in a text file, it needs to end with a newline character. (See here).

Solution

I replaced this:

json.dump(json_data, open("foobar.json", "w"), indent=4)

with this:

with open("foobar.json", "w") as json_file:
    json_text = json.dumps(json_data, indent=4)
    json_file.write("{}\n".format(json_text))  # Add newline cause Py does not
Advertisements

How to build Caffe

Building Caffe from source is easy. There are two options provided: using a Makefile or using CMake. I found that using Makefile was easier, since CMake was erroneously complaining about -lpthreads (which is a Clang library, not a GCC library).

  • Download or clone the source code of Caffe from here.

  • Copy the Makefile config file, open it in an editor and check whether all the paths are set correctly:

$ cp Makefile.config.example Makefile.config
$ vim Makefile.config
  • To build Caffe:
$ make all

MatCaffe

Building support for MATLAB to use Caffe takes a few more steps:

  • In Makefile.config, set the MATLAB_DIR to the path that contains bin/mex. On my computer, this was the path /usr/local/MATLAB/R2014a.

  • Compiling the MATLAB wrapper gave an error about C++11 on my Ubuntu 14.04 due to the older GCC compilers. So, I had to add -std=c++11 to the CXXFLAGS variable in Makefile as shown here.

  • Finally, compiling MatCaffe is easy:

$ make all matcaffe

Could not create logging file error

Problem

I tried to train a model using Caffe on a remote computer. The training proceeded without any problems, but this error would show up in the console output occasionally:

Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170221-111719.33617!I0221 11:17:19.399444 33617 solver.cpp:243] Iteration 86860, loss = 3.38734

Solution

This error turned out to be from GLog, the Google logging module. Caffe uses GLog for its logging. On this particular remote computer, the GLOG_log_dir environment variable had been set, but to a non-existent path! This was the reason GLog was complaining. Once I removed this environment variable and restarted the training, I no longer saw the error.

Tried with: Ubuntu 14.04

ImageMagick display

ImageMagick display

The display program of ImageMagick is useful when you are at the shell and need to quickly view one or a few images. It can also be used to apply many of the image processing operations available in ImageMagick on an image and view it. This viewer is extremely light and great for checking some images quickly on a remote computer over SSH.

  • If you do not have ImageMagick, installing it is easy:
$ sudo apt install imagemagick
  • To view a specific image:
$ display foo.jpg
  • To view all the JPEG files in current directory:
$ display *.jpg
  • To display all image files in current directory:
$ display *
  • If you know that you will not be changing the images, but just viewing them, then open them in immutable mode:
$ display -immutable *.jpg
  • Press F1 to view help about the mouse operations and keyboard operations available in this program.

  • Some of the keyboard shortcuts I frequently use:

Space: Next image
Backspace: Previous image
Esc: Quit
/: Rotate by 90 degrees
<: Zoom out
>: Zoom in

Tried with: ImageMagick 6.8.9.9 and Ubuntu 16.04

How to install Ubuntu on computer with UEFI

  • Unified Extensible Firmware Interface (UEFI) is a modern replacement over the BIOS for booting up operating systems. In most BIOS settings, you can choose whether you want UEFI enabled or disabled.

  • Ubuntu works best with BIOS, not UEFI. So, if you are only installing Ubuntu on a computer or you have Windows 7 or older on it then you can disable UEFI in the BIOS settings and proceed to install Ubuntu with a / partition and a swap partition.

  • However, if you already have Windows 8 or 10 installed on the computer, it will be using UEFI to boot and so UEFI cannot be disabled. Also, if you intend to install and dual boot with Windows in the future, it may be a good idea to keep UEFI enabled.

  • If you already have Windows 8 or 10 installed then you will notice an extra 100MB partition named EFI while installing Ubuntu. Do not touch it and proceed with creating a / and swap partition. Ubuntu will automatically install GRUB to use that EFI partition for booting.

  • If you do not have Windows, but want to keep UEFI enabled, then you will need to create a 100-200MB partition and pick its type as boot EFI while installing Ubuntu. The rest of the partitions for Ubuntu are the same. Just remember to create an EFI partition.

  • If you forgot to create an EFI partition on a UEFI enabled computer and installed Ubuntu, then the installer will fail at the end with this error:

The `grub-efi-amd64-signed` package failed to install into / target/.
Without the GRUB boot loader, the installed system will not boot.

Tried with: Ubuntu 16.04

How to make Intel Wireless-AC 3165 work in Ubuntu

Problem

I installed Ubuntu 16.04 on a new notebook that had Intel Wireless-AC 3165 hardware to provide wifi and Bluetooth connectivity. This hardware was working without any problems in Windows 10, without requiring any extra software installation. However, the wireless and Bluetooth features did not work in Ubuntu.

Solution

There are many solutions offered on the web to enable wireless connectivity. Copying the ucode files provided by Intel here to /lib/firmware definitely did not work.

I was using Ubuntu 16.04, which was using the Linux 4.4.x kernel. One of the suggestions online is that newer kernels have better support for Intel wireless hardware.

Ubuntu developers provide installers of all newer Linux kernels. So I decided to try my luck with the latest 4.9.x series and installed it as described here. I restarted the notebook and had wireless and Bluetooth working like magic!

How to update Linux kernel in Ubuntu

Every version of Ubuntu LTS sticks to a particular version of the Linux kernel. For example, Ubuntu 16.04 sticks to the 4.4.x series of kernels. Over the months and years you use and update this version of Ubuntu, only newer minor versions of the kernel are updated to maintain stability.

However, there might be reasons you might want to upgrade to a major new Linux kernel version. For example, to get support for newer hardware and firmware. For such options, Ubuntu developers maintain a series of the Linux mainline kernel which can be downloaded and installed.

  • Go to http://kernel.ubuntu.com/~kernel-ppa/mainline/ and decide which version of kernel you want. For example, I decided to upgrade from 4.4.x to 4.9.x.

  • You need to download 3 deb files for a full kernel installation. These will be named in this format: linux-headers-XXX_amd64.deb, linux-headers-XXX_all.deb and linux-image-XXX-generic_XXX_amd64.deb.

  • Install them:

$ sudo dpkg -i *.deb
  • Restart the computer and check if you are using the new kernel:
$ uname -r
  • Do not delete the kernel provided by Ubuntu even if apt keeps reminding you that you do not need it! If something goes wrong with the new kernel, you might want to keep the older one, so you can boot using that in the GRUB boot screen.

GParted block size warning

Problem

I opened GParted to look at the partition information of my drives. It popped up this strange warning:

The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes

I figured out that this error was about a USB thumb drive that was plugged into my computer. That USB drive was working fine and I could mount and read its files. So, what is this warning and how to fix it?

Solution

Turns out that the block size for this drive was set to 2048 bytes. It should have been 512 bytes. Its disk size was also appearing 4 times larger in GParted!

On thinking further about this, I remembered that this drive had been formatted as a bootable media to install Ubuntu using GNOME Disks as described here. That disk writer must have erroneously set the block size of this drive as 2048 bytes.

Changing back the block size to 512 was easy:

$ sudo dd if=/dev/zero of=/dev/sdc bs=2048 count=1

Replace /dev/sdc with the device ID for your USB thumb drive.

Tried with: Ubuntu 16.04

Blank display on installing Kubuntu

Problem

I had a computer with Ubuntu 16.04 running the Unity desktop. I wanted to switch to Kubuntu, so I installed it using this command:

$ sudo apt install kubuntu-desktop

Everything installed fine, except this error at the end:

kde-telepathy-minimal : Depends: kde-config-telepathy-accounts (>= 0.9.0) but it is not installed

After this, there is an error about kaccounts-provider whenever I login into Unity. If I try to login into the KDE desktop, I get a blank display!

Solution

This error seems to be hitting lots of people as seen here. This seems to be caused by a conflict between packages required by Unity and KDE. Apparently, Unity (Ubuntu) and KDE (Kubuntu) cannot be installed on the same computer!

Following some clues given by other users, I forcefully removed Telepathy since I did not need it anyway:

$ sudo dpkg --remove kde-telepathy
$ sudo apt update
$ sudo apt dist-upgrade

After this the Unity error on login and the black display in KDE went away. I was able to switch to using KDE as my primary desktop.

So, if you intend to use Kubuntu, you are better off installing a Kubuntu ISO directly instead of installing it from inside Ubuntu.

ACPI_BIOS_ERROR on installing Windows 8

Problem

I had a new notebook computer with no operating system installed on it. I had a Windows 8 license and a Windows 8 ISO which I wanted to use to install on this notebook. I prepared a Windows installer USB thumbdrive using the ISO and booted the notebook from it.

The computer would throw this error and keep restarting:

Your PC ran into a problem and needs to restart.
If you'd like to know more you can search online later for this error: ACPI_BIOS_ERROR

Solution

Searching online for this error, I found that this occurred to most people when they were updating or upgrading. Mine was neither, I was installing on a fresh computer. In any case, their solutions were either inapplicable or did not work.

While looking up solutions for this error, I discovered that I could use my Windows 8 license to install the newer Windows 10. I decided to give this a try, since Windows 10 ISO is easily available for download here. I made a Windows installer USB thumbdrive and surprise, it booted up normally and I was able to install Windows 10 Pro without any further problems!