VisIncr plugin for Vim

VisIncr

One of the most powerful aspects of using Vim is that you never have to do any drudge work. All boring repeatable tasks can be executed with a meaningful command. A recent problem I ran into was to create variable names with a large number of incrementally increasing indices. I found an awesome plugin by Dr. Chip called VisIncr that automates this very useful operation.

  • Install: Use your favorite Vim plugin installation method to install this plugin from this URL: https://github.com/vim-scripts/VisIncr

  • Usage: Let us assume I want to generate variables ranging from x_3 to x_16. I first type x_3 and repeat it 13 times one on each line. Mark the column containing 3 using visual block (Ctrl+V). Then press I and press Enter. Boom! See the output in above image. The command I is easy to remember cause the operation is i-ncrement.

  • You can refer to the documentation for the commands to increment with right justification (default is left), by a different increment or want to increment octal number, Roman numerals or dates. Pretty much any scenario that appears in code is covered by the plugin.

Tried with: Vim 7.4 and Ubuntu 16.04

H5Dump

H5Dump

HDF5 is the most popular file format for storing binary data structures. The HDFView tool makes it convenient to view and browse the contents of a HDF5 file in a Java GUI. But what if you are at the commandline and need a way to quickly view or dump the contents of a HDF5 file? The h5dump does exactly this.

  • It can be installed easily:
$ sudo apt install hdf5-tools
  • To dump the contents of a file:
$ h5dump foobar.h5
  • By default, you will notice that some data like string arrays are shown in a matrix format. To display all contents in one column format:
$ h5dump --width=1 foobar.h5
$ h5dump -w 1 foobar.h5

Tried with: HDF5 Tools 1.8.16 and Ubuntu 16.04

2016 at CodeYarns

Yet another year has passed by! 2016 was a huge year in my personal life and so I could not write as many posts as I wanted to here on my blog. Also, keeping a Twitter account is well known to cut down on your energy to write blog posts. Here are some stats from the year for CodeYarns.

20170103_stats

  • Posts: I wrote 165 posts for the blog this year. This is a big drop from 2015, when I wrote 389 posts.

  • Views: The blog saw more than 1 million views this year, which is a bit less than last year. I attribute this to the change in the name (from ChooruCode) and the corresponding change in the domain name. The views came from 830K visitors, for a views per visitor ratio of 1.26. This ratio has also been reducing over the years, which is a general trend for blogs as far as I know.

20170105_map

  • Countries: The top 3 nations of visitors continue to be USA, India and Germany. There is a long tail of nations, including 14 views from North Korea! The blog saw visitors from a total of 221 countries, which is almost all of them. The only nations which were missing in the above map seemed to me as Chad and Central African Republic.

For 2017, I hope to write an average of one post per day. Got to see if I can manage that! 🙂

Newline at end of file

Many Linux tools and compilers that process text files require the text file to end with a newline. This is simply because they are designed to process a series of lines, each ending with a newline. That means, the last line of the file should also end in a newline.

If the file does not end with a newline, then C++ compilers and other tools complain with errors such as No newline at end of file.

How do you end up with a file that does not end in newline? If you use tools that are written for other platforms like Windows or are cross-platform, those might not follow the rule of ending the last line with a newline. The latest culprit who I found doing this mistake was the Visual Studio Code editor. Editors like Vim, which come from a Unix heritage, just do not allow you to save without a newline.

In Visual Studio Code, I set the files.insertFinalNewline option to true to fix this mistake.

Reference to pointer in C++

Reference to pointer is a useful construct to be aware of in C++. As you might already know, the C++ language allows you to take a reference to any object that is not a temporary. You can take a const reference to any object, including a temporary. So what is special about reference to a pointer?

A common construct in C++ is to receive a pointer to a pointer as input argument to a function. This is typically used to allocate a primitive or an object inside the function. The allocated primitive or object will be available in the caller after the function is called and done with. This idiom is common with C programmers who have moved to C++. It is such code that becomes a lot cleaner and easier to write and read if a reference to pointer is used as input argument to function. As a bonus, since it is a reference it has to always refer to a pointer that actually exists.

The code example below shows the difference between pointer to pointer and reference to pointer:

Note how reference-to-pointer is cleaner to understand both at the caller location and inside callee.

Always make base class destructor as virtual in C++

TLDR: The title of this post says it all!

If you are having a class hierarchy, with base class and derived classes, then try to always make the base class destructor as virtual.

I recently noticed an application having a serious memory leak after merging some code. Other than the leak, everything else about the code was executing fine! After debugging the code, the culprit turned out to be a base class destructor that was not virtual. If only the above rule had been followed diligently, the error would have been caught easily.

Why this rule? The reason for this rule is pretty simple. A derived class destructor might be deallocating objects or freeing memory that it had allocated earlier during its creation or execution. Now think about the scenario where this derived class object is held using a base class pointer and it is freed.

  • If base class destructor is not virtual: Only the base class destructor is called, thus causing a memory leak.

  • If base class destructor is virtual: The derived class destructor is called first (thus freeing its allocated objects correctly) before the trail of destruction heads up the chain of hierarchy, ending in the base class destructor. This is the intended correct behavior.

Here is a code example that illustrates this scenario:

GSmartControl

20161225_gsmartcontrol

GSmartControl is a GUI program that uses the programs in smartmontools to show the SMART data of your hard disk or SSD and to run tests on it.

  • Installing it is easy:
$ sudo apt install gsmartcontrol
  • When you invoke the program, it requires superuser privileges. After starting it shows all the drives connected to your computer. You can right-click on any drive to view SMART data or run tests on it.

  • Sometimes SMART may not even be enabled on a drive. You can right-click on a drive to Enable SMART or to Enable auto offline data collection.

  • Right-click and choose View Details to get all the information about the drive.

  • Identity: This shows information about the drive like model, device, serial number and such.

  • Attributes: This shows the all-important SMART data values. The killer feature of GSmartControl is that this display is color coded. For example, values that indicate that the drive has started to degrade or fail will be in pink or red. This is a huge plus over the SMART information shown by GNOME Disks program where you have no idea what to make of the displayed values. Hovering the mouse cursor over any value gives a longer description about the value in a tooltip. This is again a huge plus since these values cannot be readily understood by the normal user.

  • Perform tests: This is the tab from where you can perform tests created by the disk manufacturer to check the health of the disk. Typically, there will be a Short test which takes ~2 minutes and an Extended test that might take an hour. Sometimes, there might be other tests offered by the disk manufacturer like a Conveyance test and so on. Running a short test should already be enough to point out if your disk is in danger.

  • If you cannot view any SMART information for the disk, then the device type that smartctl is detecting for it might be wrong. Specify a device type that matches the disk in Options → Preferences → Smartctl invocation → Smartctl parameters.

Tried with: GSmartControl 0.8.7 and Ubuntu 16.04

How to use smartctl

smartctl is a commandline tool to view S.M.A.R.T. information of a storage device and to run SMART tests on it. This is the tool that is used by other GUI tools like GNOME Disks or GSmartControl.

Installation

Installing it is easy:

$ sudo apt install smartmontools

Device information

  • To print information about a storage device using its device path:
$ sudo smartctl --info /dev/sdb
  • Here is the output I got on my external hard disk:
$ sudo smartctl --info /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Momentus 5400.7
Device Model:     ST9500320AS
Serial Number:    5WX2QYSE
LU WWN Device Id: 5 000c50 037f547bc
Firmware Version: 0002BSM2
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5799 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 1.5 Gb/s
Local Time is:    Sun Dec 25 18:29:28 2016 SGT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
  • Note the last two lines. They indicate that this storage device has SMART support and that it is enabled.

  • If the above command cannot print any SMART information, then it means that the program could not guess the device type correctly. In such a cause you will need to try some of the possible device types (see the -d option in manpage for this) and try each of them. For example, I could not view SMART info for a Toshiba Canvio USB 3.0 drive. But, I could view the SMART info when I used this command:

$ sudo smartctl -d sat --info /dev/sdb

If a certain device type works, then you must remember to use that option all the time for this disk.

  • If nothing works and you have been noticing errors on this hard disk, then it might already be too faulty to be diagnosed. It might be time to check if it can be returned for a replacement or else be discarded.

Enable or disable SMART

I have come across drives where SMART is not turned on, so you may have to manually enable it.

This is what I see when I enable SMART:

$ sudo smartctl --smart=on /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

Once enabled, SMART should remain enabled through power-cycling. Use off instead of on to disable SMART. I cannot think of any reason why you would disable it cause keeping it enabled does not affect disk performance.

Enable or disable automatic offline test

This is a test that SMART can perform every four hours to check for disk defects. Enabling this test can degrade disk performance and is obsoleted. Most recent disks will not enable this test even if you try to enable it. It is better to disable it in case it is enabled.

This is what I see when I disable it:

$ sudo smartctl --offlineauto=off /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Automatic Offline Testing Disabled.

Capabilities

You can query and view the capabilities of the storage device. This includes the self-tests that SMART of this device can perform and the estimated time they will take.

This is what I see when I view capabilities of my device:

$ sudo smartctl --capabilities /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x73) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 148) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x103b) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

For example, note how this device is telling me that performing a short self-test is estimated to take 1 minute.

Attributes

You can view the SMART attributes supported by the device and their values. With my device I get this output:

$ sudo smartctl --attributes /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   092   092   006    Pre-fail  Always       -       104646032
  3 Spin_Up_Time            0x0003   097   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       621
  5 Reallocated_Sector_Ct   0x0033   045   045   036    Pre-fail  Always       -       1128
  7 Seek_Error_Rate         0x000f   062   060   030    Pre-fail  Always       -       1606206
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       319
 10 Spin_Retry_Count        0x0013   100   099   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       393
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       445
188 Command_Timeout         0x0032   100   092   000    Old_age   Always       -       8590065680
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   066   050   045    Old_age   Always       -       34 (Min/Max 25/34)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       257
193 Load_Cycle_Count        0x0032   098   098   000    Old_age   Always       -       5170
194 Temperature_Celsius     0x0022   034   050   000    Old_age   Always       -       34 (0 20 0 0 0)
195 Hardware_ECC_Recovered  0x001a   049   046   000    Old_age   Always       -       104646032
197 Current_Pending_Sector  0x0012   095   095   000    Old_age   Always       -       103
198 Offline_Uncorrectable   0x0010   095   095   000    Old_age   Offline      -       103
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
254 Free_Fall_Sensor        0x0032   100   100   000    Old_age   Always       -       0
  • This table shows a list of attributes that the SMART of this device supports. By looking at the values in the ID column, we can see that a particular device will only support a small subset of all the attributes defined in the SMART standard.

  • The attribute Type indicates whether the value of that attribute indicates imminent failure, like in next few hours (if it says Pre-fail) or indicates that the disk is getting old (if it says Old-age).

  • The Value is a normalized value that has been converted from the Raw Value by the device firmware. Devices do not necessarily have to provide these values in a way that make sense to normal users. What we can do is to compare the value to the Threshold value and come to a decision.

  • Threshold is a value in the range 0-255. The most confusing part is that the attribute is said to have failed if the normalized value is less than the threshold (and not greater than as I assumed)! I know it does not make any sense, but that is how it is defined.

Self tests

I found that the attribute values cannot always indicate there is a problem with your disk. For example, the above values are seem normal, but this disk already has bad sectors and is failing. The best way to be sure is to run the SMART self-tests available in the device firmware. These tests are designed such that they can be performed even if the device is mounted and being used. You might notice some IO performance degradation while the test is running.

  • Here is how I ran a short self-test on my device:
$ sudo smartctl --test=short /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 1 minutes for test to complete.
Test will complete after Sun Dec 25 19:50:22 2016

Use smartctl -X to abort test.

The command returns immediately, because the self-test will be run by the device in the background.

  • If you just want to see if the self-test succeeded or failed, then query the capabilities and look at the value of the Self-test execution status.

Here is what I see on a good drive:

$ sudo smartctl --capabilities /dev/sda

Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.

Here is what I see on a failing device:

$ sudo smartctl --capabilities /dev/sda

Self-test execution status:      ( 121) The previous self-test completed having
                    the read element of the test failed.
  • To see the error log of the self-test, I use:
$ sudo smartctl --log=selftest /dev/sdb

smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-47-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%       319         849336942
# 2  Short captive       Completed: read failure       90%       319         849336942
# 3  Short offline       Completed: read failure       90%       319         849336942
# 4  Short offline       Completed: read failure       90%       319         849336942
# 5  Short offline       Completed: read failure       90%       318         849336942
# 6  Short offline       Completed: read failure       90%       318         849336942
# 7  Short offline       Completed: read failure       90%       318         849336942
# 8  Conveyance offline  Completed: read failure       90%       318         849336942

We can see that there are failures, indicating that the disk is already failing.

Tried with: smartmontools 6.4 and Ubuntu 16.04

Seagate Seatools

Seatools is a tool that can be used to test and diagnose internal and external harddisks manufactured by Seagate. It can be installed and used on Windows. There does not appear to be a Linux version available.

  • Seatools can be downloaded from here and installed easily on Windows.

  • Connect the harddisk you want to diagnose and then run Seatools. If you cannot find it in the Start menu, just check C:\Program Files (x86)\Seagate\Seatools for Windows for an executable to run.

  • When it starts it takes a lot of time looking for ATA, SCSI and USB drives connected to your computer.

  • After it starts, it shows the drives it has detected. For my external USB drive, it showed up both as the external USB drive and as an internal drive. I guess this external drive was actually created from an older internal drive series or something like that.

  • Select the drive and choose from the above menu to get its information or run short tests. My drive failed in the short test itself.

  • If the drive fails any test, it is probably unrepairable and needs to be returned (if under warranty) for a replacement or just discarded. Seatools allows you to enter the Serial and Product numbers and checks if the drive is in warranty for you.

  • From the Seagate website you can create a return order which Seagate calls RMA and then return the device using the order and labels it provides.

Tried with: Seagate Backup Plus Portable Drive 1TB

Wolfram Alpha

Wolfram Alpha is an online query system to answer any question. I only use it sometimes to query for simple mathematical objects without having to resort to the paid Mathematica software.

  • Plot: The plot query can be used to plot functions of one and two variables as 2D and 3D graphs respectively. For example, plot sin x plots a 2D graph of that function. Similarly, plot sin x cos y plots the 3D graph of that function. The 3D graphs are quite beautifully plotted with colors and contour lines, but are only shown from two views (angled and top). If you need to interactively move it around to observe it, you need to pay for a Pro version.