How to create GIF from images using ImageMagick

ImageMagick is truly a magical tool for images. It makes it super-easy to create an animated GIF file from a series of image files.

To create an animated GIF from 3 input images with a delay of 2 seconds between each frame and loop infinitely:

$ convert -delay 200 -loop 0 1.png 2.png 3.png out.gif

Tried with: ImageMagick 6.8.9.9 and Kubuntu 16.04

Advertisements

How to grep in Vim

When you are editing files in Vim, you might feel like finding out which other files have a certain text. Vim has such grep features built into it to support this usage. There are many popular Vim plugins that offer similar features, but you might want to first examine what Vim grep offers. In my opinion, most users should find the grep features built inside Vim to be quite adequate for their use.

Right off the bat, you need to know that there are two types of grep that are available in Vim: internal grep and external grep.

And you should remember that all grep operations operate from the present working directory (PWD) of Vim. You can check what is the PWD using the command :pwd. So, it is to your advantage to invoke Vim from the base directory of a code base.

Internal grep

This is the grep feature that is implemented inside Vim. Not surprisingly, this internal grep is far easier to use compared to Unix grep. However, it is also noticeably slower compared to Unix grep, for obvious reasons. However, the ease of use of internal grep is so high that I tend to prefer it over external grep.

The command to invoke internal grep is :vimgrep. But, you can use the short form :vim. Yeah this is the one command that won the Vim lottery to have the same name as the editor! 😄

This command in its most common form takes two arguments: the pattern to search for and which files to search for.

  • To search for Xanadu in all the files in the PWD: :vim Xanadu * This shows all the search results in a Quickfix window at the bottom and Vim jumps to the first match right away. You can navigate through this window and press Enter on search results to jump to them.

  • To recurse the directories or files in Vim, you have this strange syntax called star-star. For example, to search recursively in all files and directories in PWD: :vim Xanadu **

  • Once you are comfortable with the star-star syntax, you will find it very convenient to use: :vim Xanadu ../include/**/*.h

  • One trick I like to use a lot is to search for the current word under the cursor in the files. I first press * so that the word is selected. Then I use this empty pattern to search for it: :vim // *.cpp

External grep

External grep is nothing but Vim invoking the commandline grep in a shell child process. This is super dumb: Vim just invokes whatever you pass it verbatim and grabs the grep output, parses it and fills it in the Quickfix window.

  • To search in all files in the PWD: :grep Xanadu *

  • To search recursively in all files and directory under the PWD: grep -R Xanadu * Note how the recursive search is far easier to invoke in the internal grep.

Quickfix window trick to browse code

No matter whether you use internal or external grep, its results are filled in the Quickfix window. As you do new searches, new Quickfix windows are created and the latest one is displayed at the bottom. You can move through this list of Quickfix windows using the :colder and :cnewer commands.

With these grep features and Quickfix window history Vim can be used as an rudimentary IDE to examine a code base: use the grep commands to search for text (variables, methods or classes) and jump to them. And you can move back and forth in this investigation using the Quickfix window commands described above.

Tried with: Vim 7.4 and Kubuntu 16.04

Grep cheatsheet

Grep is the quintessential text search tool of the Unix shell. Many text search tools like ack and ag are popular now for searching in source code. But, for most common scenarios grep can still be a fast and good enough solution. It is available on every Linux machine you will be working at, so it is a huge bonus to be aware of its usage and capabilities.

  • Trivia: If you type :help grep in the Vim editor, you will see the origin of the name grep:
[Unix trivia: The name for the Unix "grep" command comes from ":g/re/p", where
"re" stands for Regular Expression.]
  • The most important use of grep is to search for text from standard input. This is usually done by piping output of another command to grep and searching in that. For example:
$ ls -1 | grep Xanadu

Note that the search text can be a regular expression. You can find more info about regular expressions from other detailed sources.

  • The other common use of grep is to search for text in files. For example:
$ grep Xanadu *.txt

Note that the *.txt is expanded by the shell and those filenames are passed to grep. Grep does not lookup the filenames.

  • To ignore case of the search text (make it case insensitive), use the -i option:
$ grep -i Xanadu *.txt
  • Since the filename lookup is actually done by the shell, it is not recursive by default. To search recursively in all files and directories starting from current directory:
$ grep -R Xanadu *

Note again, that the wildcard is expanded by the shell and passed to grep. Grep then takes each of those directories and it recurses through them on its own.

  • To exclude certain files, say save files, from grep use the --exclude option:
$ grep --exclude="*~" Xanadu *

To exclude multiple types of files, specify multiple --exclude options.

  • To exclude certain directories, say Git directories, from grep use the --exclude-dir option:
$ grep --exclude-dir=".git" Xanadu *
  • To exclude files based on glob patterns, you can also put those patterns in a file and pass it to the --exclude-from option. However, this did not seem to work for me.

  • By default, grep prints the file paths which have the search text. To print the line number, along with the file paths, use the --line-number option:

$ grep --line-number Xanadu *
  • Grep throws out a wall of text where the searched text can be hard to find. Asking it to use color using --color can be an useful option:
$ grep --color Xanadu *

On my computer, this shows the file path in purple, the line number in green and the searched text in red.

  • When searching text in source code files, it is helpful to get some context of where the text is found. Using the -C option you can ask grep to show you some lines of context around the result lines. For example, to get 3 lines of context above and 3 lines of context below every result line:
$ grep -C 3 Xanadu *.cpp
  • Your shell expands both matching filenames and directories and passes them as input to grep. When grep gets a directory as input, it warns that it Is a directory. For other files it might also give warnings such as No such device or address or Permission denied. To hide such messages use the --no-messages option:
$ grep --no-messages Xanadu *
  • Grep does look through binary files by default. If the searched text was found, it will just report that the binary file has that text, it does not try to print the matching line, for obvious reasons (no line separators in binary file). However, if you do not want grep poking around in binary files, use the -I or --binary-file=without-match option:
$ grep --binary-file=without-match Xanadu *

Tried with: Grep 2.25 and Ubuntu 16.04

How to debug Caffe

Caffe is written in C++ while its GPU accelerated portions are written in CUDA. It also ships with PyCaffe, a Python wrapper to the C++ code.

To debug Python code of PyCaffe

You might have written a Python script to train or load and use models for inference. The Python code of Caffe can be debugged as usual using the PDB Python debugger as described here.

To debug C++ code from Caffe binary

We can use GDB to debug the C++ code in Caffe. First, remember to build Caffe with debugging information. This can be done as described here, but remember to indicate Debug mode to CMake, like this:

$ cmake -DCMAKE_BUILD_TYPE=Debug ..

If you are using binary Caffe, then debugging the C++ code is straightforward using GDB:

$ gdb --args ./caffe --your-usual-caffe-arguments-go-here

You can set breakpoints in any Caffe C++ code and debug using GDB as usual. For more info see my GDB cheatsheet.

To debug C++ code from PyCaffe

If you are using PyCaffe and need to debug the C++ parts of Caffe code that is still possible using GDB! Remember to first build Caffe in Debug mode as shown above. Note that after make install, you might have to rename the _caffe-d.so file to _caffe.so to be able to import caffe in your Python script.

To be able to debug the C++ code, we invoke GDB and pass it the python interpreter and its arguments, which is our Python script that calls PyCaffe:

$ gdb --args python my_script_calls_pycaffe.py --some_input_arguments

GDB first begins in the Python interpreter binary. Python will later load all the required shared libraries including _caffe.so and more importantly libcaffe-nv.so which is the compiled version of the Caffe C++ code. The problem is that these libraries are not yet loaded at the beginning. What we can do is set one or more breakpoints at the places we want right away. GDB will complain that these breakpoint locations are as yet not known to it. That is okay, it will turn them into pending breakpoints that will be enabled when the shared library having that location is loaded.

For example, to set up a breakpoint right away in convolution layer code:

(gdb) b base_conv_layer.cpp:15

After you have set your breakpoints, press c to continue and GDB will load the shared libraries and stop when your breakpoint locations are hit. After this point it is debugging as usual in GDB.

If you are adventurous, you can even connect GDB to the Python process that is running PyCaffe/Caffe by using its PID. For example, after I find out that my Python script is running as PID 589:

$ gdb -p 589

You can set breakpoints in Caffe C++ code and GDB will stop at those locations.

I hope you have fun stepping through and exploring Caffe code! 🙂

Tried with: GDB 7.7.1 and Ubuntu 14.04

How to group windows as tabs in KDE

KDE/Plasma/KWin 4.x that ships in Ubuntu 14.04 has one amazing feature that I discovered today. I keep many XTerm, RXVT and related file explorer windows open. These were related to many servers I was SSHed to and I was looking for a way to group windows related to each server together. Guess what, KDE 4.x has a fantastic solution!

KDE 4.x allows you to group any set of open application windows together as tabs. Just right-click on the titlebar of an application and choose Attach as tab and pick the other application with which you want to group this one. These two application windows will now appear in a single window as tabs! You can repeat this operation and group together apps as tabs inside super-windows. (In older KDE, this option appears as Move window to group.) This works for any application window, they can be from different applications. What a great way to organize your windows on the desktop!

Sadly, this feature is missing in the recent KDE/Plasma/KWin 5.x series that ships in Ubuntu 16.04. It lies on the missing features list and I really hope they will reimplement this great feature.

How to view CMake cache variables

Running CMake for a project creates many variables that are required for its own operation and for the project being built. If you use the CMake GUI, then these are the variables which you see populated after you perform Configure and then Generate. When you want to diagnose a CMake build problem at the shell, it is useful to be able to view these variables.

  • To perform a CMake build and then print the cache variables:
$ cmake -L ..

Note that in the above command, I am in the build directory and the CMakeLists.txt is in the parent directory. This command builds as usual and at the end prints the CMake variables and their values.

  • To perform a CMake build and list all the cache variables, including advanced variables:
$ cmake -LA ..

Note that the number of variables that are printed can be quite high for big projects. I have seen it run into the hundreds.

  • To perform a CMake build and list all the cache variables and print the help documentation about each variable too:
$ cmake -LAH ..
  • If you have already done a CMake build and just need to print out the cache variables, add the -N option:
$ cmake -L -N

This command assumes that the current directory is the build directory containing the cache files, so no directory needs to be specified. This command is quick because it does not do a CMake build. Additionally, you can also add the -AH options as before if you want.

Tried with: CMake 2.8.12.2 and Ubuntu 16.04

How to check status of port using NMap

NMap is a classic Swiss army knife for network operations. I recently had to check if a certain port on a LAN computer was open or not. This type of diagnostic is very easy to do with NMap.

  • NMap can be installed easily:
$ sudo apt install nmap
  • To check if port 9010 was open on the medusa computer:
$ nmap -p 9010 medusa

Starting Nmap 6.40 ( http://nmap.org ) at 2017-08-25 13:05 SGT
Nmap scan report for medusa (192.168.110.22)
Host is up (0.00035s latency).
PORT     STATE SERVICE
9010/tcp open  sdr

Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds

Turns out that the port 9010 was open.

  • To check port 9999 on medusa, which was actually closed:
$ nmap -p 9999 medusa

Starting Nmap 6.40 ( http://nmap.org ) at 2017-08-25 13:06 SGT
Nmap scan report for medusa (192.168.110.22)
Host is up (0.00030s latency).
PORT     STATE  SERVICE
9999/tcp closed unknown

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds
  • To check the status of ports in the range on the medusa computer:
$ nmap -p 0-10 medusa

Starting Nmap 6.40 ( http://nmap.org ) at 2017-08-25 13:04 SGT
Nmap scan report for medusa (192.168.110.22)
Host is up (0.00027s latency).
PORT   STATE  SERVICE
0/tcp  closed unknown
1/tcp  closed tcpmux
2/tcp  closed compressnet
3/tcp  closed compressnet
4/tcp  closed unknown
5/tcp  closed unknown
6/tcp  closed unknown
7/tcp  closed echo
8/tcp  closed unknown
9/tcp  closed discard
10/tcp closed unknown

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

Tried with: NMap 7.01 and Ubuntu 16.04

ImportError on tilde path

Problem

I had a Python module named foobar.py located inside ~/foobar directory. To be able to import it, I did this in my Python code:

import sys
sys.path.append("~/foobar")
import foobar

However, when that script was run, it gave this error:

Traceback (most recent call last):
  File "./hello_world.py", line 10, in 
    import foobar
ImportError: No module named foobar

Solution

It turns out that Python cannot expand the tilde to my home directory! This was quite surprising to me. Once I expanded it to the absolute path, the import worked fine.

Tried with: Python 2.7.12 and Ubuntu 16.04

AdvancedSorters plugin for Vim

Vim has an inbuilt :sort function that can be used to sort lines. However, I recently had to sort words in a line. I could have broken the line and put each word on one line, sort them and put the results back as a line. Thankfully, I found the AdvancedSorters plugin for Vim that does this job easily in one command.

Tried with: Vim 7.4 and Ubuntu 16.04