Python JSON dump misses last newline

Problem

The dump method from the Python json package can be used to write a suitable Python object, usually a dictionary or list, to a JSON file. However, I discovered that Unix shell programs have problems working with such a JSON file. This turned out to be because this dump method does not end the last line with a newline character! According to the POSIX definition of a line in a text file, it needs to end with a newline character. (See here).

Solution

I replaced this:

json.dump(json_data, open("foobar.json", "w"), indent=4)

with this:

with open("foobar.json", "w") as json_file:
    json.dump(json_data, json_file, indent=4)
    json_file.write("\n")  # Add newline cause Py JSON does not
Advertisements

Newline at end of file

Many Linux tools and compilers that process text files require the text file to end with a newline. This is simply because they are designed to process a series of lines, each ending with a newline. That means, the last line of the file should also end in a newline.

If the file does not end with a newline, then C++ compilers and other tools complain with errors such as No newline at end of file.

How do you end up with a file that does not end in newline? If you use tools that are written for other platforms like Windows or are cross-platform, those might not follow the rule of ending the last line with a newline. The latest culprit who I found doing this mistake was the Visual Studio Code editor. Editors like Vim, which come from a Unix heritage, just do not allow you to save without a newline.

In Visual Studio Code, I set the files.insertFinalNewline option to true to fix this mistake.

A strange error on running Python script at the shell

Problem

I have a Python script named foo.py. Since, I would like to run it directly from the shell, I have added a shebang to run it through Python:

#!/usr/bin/env python
print "Hello world"

If I run it directly at the shell, I get a strange error:

$ ./foo.py
: No such file or directory

However, if I run it explicitly with the Python interpreter, it works fine:

$ python foo.py
Hello world

Solution

This strange error turned out to be due to the newline markers in the file. The file was created on Windows and had CR+LF as the newline marker. But, I was running the script on Linux, which only uses LF as the newline marker.

So, this must have led the env program to run the character CR as a command. And obviously, it could not find any such file or directory.

The solution is simple. Convert the newline markers in the file to that of the host system, in this case to Linux. This can be done easily in Vim by using the fileformat option.

Tried with: GNU CoreUtils 8.12.197, Python 2.7.3 and Ubuntu 12.04 LTS

How to view and change the newline markers of a file in Vim

It is pretty common to create, edite and exchange source files between Windows, Linux and Mac computers. The newline markers in text files are different on all these three systems. Sometimes, this leads to funny looking characters when files are created on one system and opened on another. Typically, the dos2unix and unix2dos tools are used to convert text files between these formats.

In Vim, it is easy to convert the end-of-line markers of a file to that of Windows, Linux or Mac. This is done using the fileformat option.

To view the end-of-line markers of the currently open file:

:set fileformat?

To set the end-of-line markers of the file to Windows:

:set fileformat=dos

To set the end-of-line markers of the file to Linux:

:set fileformat=unix

To set the end-of-line markers of the file to Mac:

:set fileformat=mac

After changing the newline markers, save the file.

Tried with: Vim 7.3 and Ubuntu 12.04 LTS

Why a multi-line text file appears as single line in Notepad

A text file with multiple lines of text appears concatenated into a single line when opened in NotepadΒ on Windows.

This happens because of the difference in the way the end-of-line is written by Linux (or other Unix-based OS) and Windows. If this text file was created on Linux and opened on Windows using Notepad, this problem occurs.

Linux uses the newline character (ASCII: 0x0A) to indicate end-of-line. Windows uses two characters: carriage return (ASCII: 0x0D) followed by newline (ASCII: 0x0A) to indicate end-of-line.

Almost any Windows editor other than Notepad, like Notepad++ for example, can understand both types of end-of-line indicators and can open such text files correctly.