Stub library warning on


I tried to run a program compiled with CUDA 9.0 inside a Docker container and got this error:


You should always run with that is installed with your
NVIDIA Display Driver. By default it's installed in /usr/lib and /usr/lib64. in GDK package is a stub library that is attached only for
build purposes (e.g. machine that you build your application doesn't have
to have Display Driver installed).


Let us first try to understand the error and where it is coming from. The program compiled with CUDA 9.0 has been linked to This is the shared library file of the NVIDIA Management Library (NVML). During execution, is throwing this error. Why?

From the error message, we get an indication that there are two files. One is a stub that is used during compilation and linking. I guess it just provides the necessary function symbols and signatures. But that library cannot be used to execute the compiled executable. If we do try to execute with that stub shared library file, it will throw this warning.

So, there is a second, the real shared library file. It turns out that the management library is provided by the NVIDIA display driver. So, every version of display driver will have its own file. I had NVIDIA display driver 384.66 on my machine and I found under /usr/lib/nvidia-384. The stub library file allows you to compile on machines where the NVIDIA display driver is not installed. In our case, for some reason, the loader is picking up the stub instead of the real library file during execution.

By using the chrpath tool, described here, I found that the compiled binary did indeed have the stub library directory in its path:/usr/local/cuda/lib64/stubs. That directory did have a Using the strings tool on that shared library, confirmed that it was the origin of the above message:

$ strings | grep "You should always run with"

Since the binary has an RPATH, described here, with the stubs path, the stub library was getting picked up with high preference over the actual, which was present in . The solution I came up with for this problem was to add a command to the docker run invocation to delete the stubs directory:

$ rm -rf  /usr/local/cuda/lib64/stubs

That way, it was still available outside Docker for compilation. It would just appeared deleted inside the Docker container, thus forcing the loader to pick up the real during execution.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.