Many libraries can only be installed by building from source and installing using
make install. Removing these installations can be quite painful.
In some rare cases, these Makefiles are written to support uninstall. In that case, you can go the same directory from where you did
make install and try
make uninstall. The only library I've seen supporting this is OpenCV.
For other installs which do not support uninstall, I highly recommend installing using checkinstall as described here. Tried with: Ubuntu 14.04