How to view PIP dependencies using pipdeptree

When you install a Python PIP package using the pip tool, you will notice that it figures out if the package depends on other packages and installs them too. It is sometimes useful to know what is this tree of packages that a package depends on. pipdeptree is a tool that makes it easy to view this information.

  • Installing pipdeptree is easy:
$ sudo pip3 install pipdeptree
  • View the dependency tree of every installed package
$ pipdeptree
  • View the dependency tree of a particular package foobar:
$ pipdeptree -p foobar
  • View the reverse dependency tree — the packages that are dependent on every installed package:
$ pipdeptree -r
  • Render the dependency tree graph to a PDF file using GraphViz:
$ pipdeptree --graph-output pdf > out.pdf

Tried with: Ubuntu 18.04

Advertisements

How to build and install TensorFlow

TensorFlow (TF) can be built from source easily and installed as a Python wheel package. I used the following steps to build it using Python3 and with support for CUDA and TensorRT:

  • Install Python3 pre-requisites:
$ sudo apt install python3-dev python3-pip
  • Install necessary Python3 packages locally:
$ pip3 install -U --user six numpy wheel setuptools mock
$ pip3 install -U --user keras_applications==1.0.6 --no-deps
$ pip3 install -U --user keras_preprocessing==1.0.5 --no-deps

These packages are installed to your ~/.local/lib/python3.x/site-packages directory. TF documentation also installs the latest pip3 from PyPI. However, doing that causes the infamous “Cannot import name main” error, so I do not do that.

  • TF uses Bazel as its build tool. Install it as described here. I ended up placing its binary in my ~/bin. Since my ~/bin is in my PATH, the Bazel binary can be executed from any place.

  • I recommend creating a tensorflow_root directory. This is because the TF packaging tends to write out to a location outside the TF source directory. Also, TF needs to access other libraries. So this root directory makes it easy to create all TF related directories under one umbrella.

  • Clone the TF Git repository inside the root directory:

$ cd tensorflow_root
$ git clone git@github.com:tensorflow/tensorflow.git
$ cd tensorflow
  • Configure the build process using:
$ ./configure

Some of the questions it asks and my replies:

  • Please specify the location of python. /usr/bin/python3
  • Please input the desired Python library path to use. /usr/local/lib/python3.6/dist-packages
  • Enable: XLA JIT, CUDA and TensorRT. Be careful, TF might not work with latest versions of CUDA, cuDNN and TensorRT. I used CUDA 10.0 and cuDNN 7.3 and TensorRT 5.0.
  • Did not enable: OpenCL SYCL and ROCm.
    • It is time to build TF. The command to build is:
$ bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

The first build can take 2-4 hours to complete.

  • Now we are ready to package the build artifacts into a Python package. Specify where you want that package to be placed in the packaging command:
$ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /path_to_tensorflow_root/tensorflow_pkg

I found that it had generated a Python wheel file named: tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl

  • We are ready to install TF. Make sure you remove older versions of TF:
$ pip3 uninstall tensorflow tensorflow-estimator
  • Install TF from the Python wheel package:
$ pip3 install -U --user tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl
  • Check if TF is installed correctly:
$ python3 -c "import tensorflow"

Tried with: Tensorflow 1.13.1 and Ubuntu 18.04

How to set environment variable in gdb

GDB inherits the environment variables from your shell. Some environment variables, like LD_LIBRARY_PATH, might not be inherited for safety reasons. This can cause errors like this when you try to debug:

$ gdb --args ./a.out
(gdb) r
/path/to/a.out: error while loading shared libraries: libfoobar.so.5: cannot open shared object file: No such file or directory

You can set the LD_LIBRARY_PATH environment variable at the gdb shell using the set env command like this:

(gdb) set env LD_LIBRARY_PATH /path/to/1:/path/to/2

However, if you use a shell like Fish, you will notice that this will silently not work and you still get the cannot open shared object file error.

This is because under the covers, gdb is reading the SHELL environment variable to understand what is the shell and doing what is right for that shell. It might not understand how to work with the Fish, which is a relatively new shell.

The solution that works for me, is to set the SHELL variable at the Fish shell to the Bash path:

$ set SHELL (which bash)

And then launch gdb and set the environment variable as shown above. It works after that.

Reference: Your program’s environment (GDB Manual)

Unicode BOM problem

Problem

I processed a JSON file using some tool and the resulting JSON text file would not be accepted by other tools. They would complain that this was a UTF-8 Unicode (with BOM) text file. I had to remove whatever this BOM was from my UTF-8 file.

Solution

BOM is a byte order mark added by some tools to UTF-8 files. BOM is this 3-byte sequence: 0xEF,0xBB,0xBF.

You could use any tool or process to remove these 3-byte sequences. If you are on Linux, the awesome sed tool can do the job:

$ sed -i '1s/^\xEF\xBB\xBF//' in.txt

Reference: How can I remove the BOM from a UTF-8 file?

GDB Colour Filter

A problem I face regularly with GDB is the backtrace. The stack trace lists out the function frames currently on the stack and it is a wall of text. it is hard to discern the function address, function name, function parameters, source file path and line number.

Normal GDB backtrace

This is precisely the problem that GDB Colour Filter solves. Clone its repo and source its Python file inside your ~/.gdbinit and you are set. Backtraces are now printed with distinctly different colors and formatting for all the components of a function frame. I especially find it useful to pick out the function name and the source file and line number.

GDB backtrace with GDB Colour Filter

There is only one slight problem: to display the components of a function frame at a consistent column this breaks down a frame into two lines. So your backtrace lines are doubled and might fill up the display when you try this.

Google Form redirection error

Problem

A team member shared a Google Form for me to fill out. Strangely that form URL could not open in Firefox. It would not load with this error:

The page isn’t redirecting properly
An error occurred during a connection to docs.google.com.
This problem can sometimes be caused by disabling or refusing to accept cookies.

Solution

I did not have any setting in Firefox to disable or refuse some cookies. So that error was wrong. But it gave a hint that the problem might be caused by cookies.

I went into the Firefox options and deleted all cookies involved with the URL google.com. There is no need to delete all cookies, just those that have this URL. Once I did that, the Google Form URL opened correctly.

How to print in a Makefile

Makefiles and especially recursive Makefiles can be hard to understand and debug. But the good old debugging method of printing out values from certain locations in the source code can be employed here too.

  • To print a message when make parses past that location in the Makefile:
$(info Hello World Make just parsed past this line)

Note that make typically parses a file twice before it executes what is needed to satisfy its targets. So, you will see the message you are printing twice.

  • To print the value of a make variable at a certain location in the Makefile:
$(info $(FOOBAR_VAR))

CDex

CDex is a Windows tool for ripping audio tracks from audio CDs to WAV or MP3 files. This is a great utility for ripping the audio CD collection gathering dust in your attic and converting them for playing on devices without a CD/DVD drive.

CDex is straightforward to use:

  • Insert your audio CD into a drive connected to your Windows computer. Its tracks will appear listed in CDex.
  • You can pull the artist/album/track info from FreeDB using the CDDB → Read remote FreeDB option. You would need to provide an email address in the Settings before this can work.
  • To rip to MP3, use the Extract CD tracks to Compressed Audio Files option on the right. This uses the LAME MP3 Encoder, which is installed along with CDex.
  • You can configure the directory and filename format for the ripped MP3 files in the settings.

That is all there is to it.

Tried with: CDex 2.16 and Windows 10 x64

Caffe CUDA_cublas_device_LIBRARY error

Problem

I was trying to build BVLC Caffe from source as described here on Ubuntu 18.04 with CUDA 10.0.

CMake ended with this error:

Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_cublas_device_LIBRARY (ADVANCED)

Solution

This problem seems to occur with a combination of CUDA 10.0 or later and a CMake version that is older than 3.13. I upgraded to CMake 3.14.0 as described here and the problem was solved.

How to install CMake

CMake is easy to install in Ubuntu using apt:

$ sudo apt install cmake

However, depending on your version of Ubuntu, the CMake version that is installed might be very old. So, you might run into problems when you build projects that use features from more recent versions of CMake.

CMake provides binary versions for Linux x86_64. Installing the latest version of CMake from these packages is easy:

  • Remove the Ubuntu version of CMake:
$ sudo apt remove cmake cmake-data
  • Download the .sh binary package of the CMake version you want from here. When I downloaded, I got cmake-3.14.0-Linux-x86_64.sh

  • Move the downloaded package to /opt and execute it:

$ sudo mv cmake-3.14.0-Linux-x86_64.sh /opt
$ cd /opt
$ sudo chmod +x cmake-3.14.0-Linux-x86_64.sh
$ sudo bash ./cmake-3.14.0-Linux-x86_64.sh

This installs this version of CMake in the directory /opt/cmake-3.14.0-Linux-x86_64.

  • Create symbolic links for the CMake binaries:
$ sudo ln -s /opt/cmake-3.14.0-Linux-x86_64.sh/bin/* /usr/local/bin

$ Test if CMake is working:

$ cmake --version
cmake version 3.14.0

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Tried with: Ubuntu 18.04