How to broadcast using wall

There are times when you wan to send a message to the users logged in and using a server machine. The classic wall command is useful for that scenario. The message you send will appear at every interactive shell of every user on the machine. The best thing about this command is that no special chat or messenger client is required.

  • Wall is a classic Unix command and should be installed as part of the bsdutils package. In case you don’t have it:
$ sudo apt install bsdutils
  • Broadcasting a message is easy:
$ wall "hello folks, server will be shutdown at 1PM"

It would have been nice if this command could be used to address one specific user or a subset of users, but that is not possible.

Advertisements

How to discover type hierarchy in Python

Given any type in Python, you can easily discover its ancestor and descendant types. This ease of discovery of the internals of the language is one of my favorite features of Python.

  • Remember that all types are descended from the object type.

  • Even type is a type and it is a child of the object type.

  • The __base__ attribute of any type has a string value with the name of the parent type.

  • The __subclasses__ method of any type lists the child types.

  • To determine which are the standard types (or builtin types or builtins as they are called in Python), check the __module__ attribute of the type. If it is builtins in Python 3 or __builtin__ in Python 2, then that is a standard type.

  • If you start from object, you can actually list the entire type hierarchy tree. A script that does just that can be found here.

  • In Python 3.5.2, I found that there are 143 builtin types (most of them are just types of Exception) in the tree:

object
+-- type
+-- dict_values
    +-- odict_values
+-- tuple_iterator
+-- set
+-- fieldnameiterator
+-- frame
+-- dict_keyiterator
+-- PyCapsule
+-- coroutine
+-- bytearray
+-- NoneType
+-- list
+-- dict
+-- getset_descriptor
+-- method-wrapper
+-- method
+-- str_iterator
+-- formatteriterator
+-- str
+-- set_iterator
+-- range_iterator
+-- memoryview
+-- cell
+-- generator
+-- map
+-- list_iterator
+-- stderrprinter
+-- reversed
+-- method_descriptor
+-- code
+-- weakproxy
+-- int
    +-- bool
+-- ellipsis
+-- module
+-- dict_items
    +-- odict_items
+-- bytearray_iterator
+-- Struct
+-- moduledef
+-- filter
+-- staticmethod
+-- tuple
+-- frozenset
+-- managedbuffer
+-- coroutine_wrapper
+-- function
+-- builtin_function_or_method
+-- odict_iterator
+-- float
+-- range
+-- super
+-- dict_keys
    +-- odict_keys
+-- list_reverseiterator
+-- bytes_iterator
+-- member_descriptor
+-- wrapper_descriptor
+-- property
+-- instancemethod
+-- zip
+-- weakref
+-- slice
+-- longrange_iterator
+-- dict_valueiterator
+-- EncodingMap
+-- callable_iterator
+-- mappingproxy
+-- BaseException
    +-- Exception
        +-- TypeError
        +-- StopAsyncIteration
        +-- SyntaxError
            +-- IndentationError
                +-- TabError
        +-- AttributeError
        +-- AssertionError
        +-- StopIteration
        +-- MemoryError
        +-- BufferError
        +-- NameError
            +-- UnboundLocalError
        +-- LookupError
            +-- IndexError
            +-- KeyError
        +-- EOFError
        +-- ImportError
        +-- ValueError
            +-- UnicodeError
                +-- UnicodeEncodeError
                +-- UnicodeDecodeError
                +-- UnicodeTranslateError
        +-- RuntimeError
            +-- RecursionError
            +-- NotImplementedError
        +-- SystemError
        +-- Warning
            +-- UserWarning
            +-- DeprecationWarning
            +-- BytesWarning
            +-- SyntaxWarning
            +-- PendingDeprecationWarning
            +-- FutureWarning
            +-- ResourceWarning
            +-- ImportWarning
            +-- RuntimeWarning
            +-- UnicodeWarning
        +-- ReferenceError
        +-- OSError
            +-- ConnectionError
                +-- BrokenPipeError
                +-- ConnectionAbortedError
                +-- ConnectionRefusedError
                +-- ConnectionResetError
            +-- BlockingIOError
            +-- NotADirectoryError
            +-- PermissionError
            +-- FileExistsError
            +-- TimeoutError
            +-- IsADirectoryError
            +-- InterruptedError
            +-- ProcessLookupError
            +-- FileNotFoundError
            +-- ChildProcessError
        +-- ArithmeticError
            +-- FloatingPointError
            +-- OverflowError
            +-- ZeroDivisionError
    +-- GeneratorExit
    +-- KeyboardInterrupt
    +-- SystemExit
+-- dict_itemiterator
+-- classmethod
+-- NotImplementedType
+-- iterator
+-- bytes
+-- enumerate
+-- classmethod_descriptor
+-- complex
+-- traceback
+-- weakcallableproxy
  • Note how bool is a child type of the int type.

  • In Python 2.7.12, I found that there are 60 builtin types in the tree:

object
+-- type
+-- weakref
+-- weakcallableproxy
+-- weakproxy
+-- int
    +-- bool
+-- basestring
    +-- str
    +-- unicode
+-- bytearray
+-- list
+-- NoneType
+-- NotImplementedType
+-- traceback
+-- super
+-- xrange
+-- dict
+-- set
+-- slice
+-- staticmethod
+-- complex
+-- float
+-- buffer
+-- long
+-- frozenset
+-- property
+-- memoryview
+-- tuple
+-- enumerate
+-- reversed
+-- code
+-- frame
+-- builtin_function_or_method
+-- instancemethod
+-- function
+-- classobj
+-- dictproxy
+-- generator
+-- getset_descriptor
+-- wrapper_descriptor
+-- instance
+-- ellipsis
+-- member_descriptor
+-- file
+-- PyCapsule
+-- cell
+-- callable-iterator
+-- iterator
+-- EncodingMap
+-- fieldnameiterator
+-- formatteriterator
+-- module
+-- classmethod
+-- dict_keys
+-- dict_items
+-- dict_values
+-- deque_iterator
+-- deque_reverse_iterator
+-- Struct
  • Note how str and unicode are child types of the basestring type. Also observe how this differs from Python 3 builtin types.

  • Also notice how in Python 2 the exception types are not builtin types.

Gramps Genealogy Program

Gramps is a great open source tool to create, maintain and visualize your family tree. I had a great time asking around my family about our ancestors and exporting beautiful ancestor tree diagrams and descendant tree diagrams.

  • You can install an old version directly from Ubuntu repositories:
$ sudo apt install gramps
  • Or you can download the latest .deb file from here and install it:
$ sudo dpkg -i gramps_you_downloaded.deb
  • When you first open it, it asks you to create a family tree. This is the document inside which you can maintain one family tree. Give it any name you wish.

  • On the left are different views into your family tree. Out of these, I found that I can get by using just People, Relationships and Charts tabs.

  • Add a person: Start off by adding entry for one person (say your earliest ancestor or yourself) by going to the People tab and then choosing + (Add a new person) option in the topbar. You can enter Given Name, Surname and in the Events section at the bottom, add a Birthday. Get used to this dialog, you will be seeing it a lot when adding or editing people to your tree.

  • Add a relationship: Click a person’s entry in the People tab and then switch to the Relationship tab to add partners (spouse) and children. Choose Edit → Add a Partner to add a spouse and their details. At the bottom of that dialog, you can add one or more Child entries.

  • Charts: Switch to Charts view to see ancestor tree of a chosen person. The default side view is quite confusing to me. I like to change the chart view by choosing View → Configure and choosing Vertical ↑ as the Tree Direction. This ancestor tree view looks more intuitive to me. You can double-click any unfilled ancestor box to add that person to the tree.

  • Ancestor tree: A beautiful ancestor tree for any person (say you or your child) can be exported to PDF. Select the person in any view and choose Reports → Graphical Reports → Ancestor Tree. Note that there is an option in there to resize the PDF to fit the entire tree. You can later convert the PDF to PNG using ImageMagick tools, like described here.

  • Descendent tree: A beautiful descendant tree for any person (say your oldest ancestor) can be exported to PDF. Select the person in any view and choose Reports → Graphical Reports → Descendant Tree.

Tried with: Gramps 4.2.6 and Ubuntu 16.04

How to read YAML file in Python with ordered keys

It is very easy to read a YAML file in Python as a combination of dict and lists using PyYAML. However, the YAML format does not require PyYAML to read the keys of any dict in the YAML file to be read in the order it appears in the file. In addition, Python dict also does not have any order to the keys in it. However, in certain situations it might be necessary to read the keys in YAML in the order they appear in the file. This can be done by using the yamlordereddictloader.

  • Installing this Python package is easy using pip:
$ sudo pip install yamlordereddictloader
  • Read YAML files by providing the loader from this package to PyYAML:
import yaml
import yamlordereddictloader

with open("foobar.yaml") as f:
    yaml_data = yaml.load(f, Loader=yamlordereddictloader.Loader)

This returns the data in the YAML file as a combination of lists and OrderedDict (instead of dict). So, almost all of the rest of the your code should work the same as before after this change.

Tried with: yamlordereddictloader 0.4 and Ubuntu 16.04

pstree cheatsheet

The ps command in Linux has been horribly mauled by having to support Unix, BSD and GNU options and innumerable eccentric back-compatible necessities that I find it almost unusable. Instead, I like to use the pstree command for almost all my process exploration tasks. pstree shows the processes on a system as a tree.

  • Installing it is easy:
$ sudo apt install psmisc
  • By default, pstree shows the processes of all users as a tree, where each link connects the parent process on the left to the child process on the right:
$ pstree
  • To only view the processes of a particular user joe:
$ pstree joe
  • To view the command-line arguments of each process:
$ pstree -a
  • To view the PID of each process:
$ pstree -p
  • To view all the children or descendants of a process with PID 900:
$ pstree 900
  • To view both the ancestors and descendants of a process with PID 900:
$ pstree -s 900

Tried with: pstree 22.21 and Ubuntu 16.04

urlopen got multiple values error

Problem

I tried to run a Python script given by a friend. It ended with this error:

  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 79, in request
    **urlopen_kw)
  File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 142, in request_encode_body
    **urlopen_kw)
TypeError: urlopen() got multiple values for keyword argument 'body'

Solution

The script used a Python package, which in turn used urllib3. This strange error has nothing to do with my code, but with the urllib3 package. The urllib3 package installed by Ubuntu was pretty old: 1.7.1. Updating to a more recent version of the package will fix the error. Either upgrade using sudo pip install urllib3 for system-wide update or update inside your virtualenv.

xdiskusage

The du command is a useful tool to check the size of directories and files on a filesystem or inside a directory. However, its output is not the most intuitive, especially if you want to probe further into child directories trying to find what is occupying space. There are ncurses console tools like ncdu and GUI tools like Baobab that help with this. Another good alternative I have discovered is xdiskusage.

xdiskusage is a GUI application that parses the output of du to show the disk space occupied by directories and files. Written in the FLTK toolkit, it is super-light and is possibly the lightest and fastest disk usage tool I have used. That makes it great for using over SSH sessions too. It uses the treemap visualization, so it is much easier to investigate disk usage than looking at du output.

  • Installing it is easy:
$ sudo apt install xdiskusage
  • Invoke without any arguments if you want to be presented with a list of all the filesystems on your computer. Double-click on any to view treemap visualization of its contents.

  • To look under a particular directory, pass it as the first argument to the program.

  • In the treemap visualization, you can keep probing further by double-clicking on any rectangle.

  • Right-click in the GUI to get menu of all possible actions using mouse. You can also see the keyboard bindings available for browsing using keyboard.

Tried with: xdiskusage 1.48 and Ubuntu 16.04

Stub library warning on libnvidia-ml.so

Problem

I tried to run a program compiled with CUDA 9.0 inside a Docker container and got this error:

WARNING:

You should always run with libnvidia-ml.so that is installed with your
NVIDIA Display Driver. By default it's installed in /usr/lib and /usr/lib64.
libnvidia-ml.so in GDK package is a stub library that is attached only for
build purposes (e.g. machine that you build your application doesn't have
to have Display Driver installed).

Solution

Let us first try to understand the error and where it is coming from. The program compiled with CUDA 9.0 has been linked to libnvidia-ml.so. This is the shared library file of the NVIDIA Management Library (NVML). During execution, libnvidia-ml.so is throwing this error. Why?

From the error message, we get an indication that there are two libnvidia-ml.so files. One is a stub that is used during compilation and linking. I guess it just provides the necessary function symbols and signatures. But that library cannot be used to execute the compiled executable. If we do try to execute with that stub shared library file, it will throw this warning.

So, there is a second libnvidia-ml.so, the real shared library file. It turns out that the management library is provided by the NVIDIA display driver. So, every version of display driver will have its own libnvidia-ml.so file. I had NVIDIA display driver 384.66 on my machine and I found libnvidia-ml.so under /usr/lib/nvidia-384. The stub library file allows you to compile on machines where the NVIDIA display driver is not installed. In our case, for some reason, the loader is picking up the stub instead of the real library file during execution.

By using the chrpath tool, described here, I found that the compiled binary did indeed have the stub library directory in its path:/usr/local/cuda/lib64/stubs. That directory did have a libnvidia-ml.so. Using the strings tool on that shared library, confirmed that it was the origin of the above message:

$ strings libnividia-ml.so | grep "You should always run with"

Since the binary has an RPATH, described here, with the stubs path, the stub library was getting picked up with high preference over the actual libnvidia-ml.so, which was present in . The solution I came up with for this problem was to add a command to the docker run invocation to delete the stubs directory:

$ rm -rf  /usr/local/cuda/lib64/stubs

That way, it was still available outside Docker for compilation. It would just appeared deleted inside the Docker container, thus forcing the loader to pick up the real libnvidia-ml.so during execution.

How to find the init system used by Linux

init is the first process that is started by an Unix operating system. It has PID 1 and handles creation of all other processes and daemon processes required by the OS. It acts as the ancestor of all processes. There are many init systems that have been used in Linux distributions over the years. Some of the popular init systems are the classic System V init, upstart (Ubuntu used to use this) and systemd (popular right now for all Linux distros).

So, you are sitting at a Linux computer, do you know which init system it is using? There is no straightforward method to find this out.

Here are a few methods, one of which should work:

  • Check the init version, it should report the init system name too:
$ init --version
init (upstart 1.12.1)
Copyright (C) 2006-2014 Canonical Ltd., 2011 Scott James Remnant

This was on a Ubuntu 14.04 system. You can see that it uses upstart.

  • Sometimes the /sbin/init is a symbolic link to the actual init:
$ stat /sbin/init
  File: '/sbin/init' -> '/lib/systemd/systemd'

This was on a Ubuntu 16.04 system. You can see that it uses systemd.