LightGBM is currently one of the best implementations of gradient boosting. I will not go in the details of this library in this post, but it is the fastest and most accurate way to train gradient boosting algorithms. And it has a GPU support. Installing something for the GPU is often tedious… Let’s try it!
Setting up LightGBM with your GPU
I will assume a nVidia GPU. I personnally have a GeForce GTX 745, with the Driver Version: 410.48. If you do not have a GPU already, be careful in the model you chose. When buying a GPU, you have to make sure the “compute capability” is high enough with respect to the software you plan to use. Per example, rapids.ai needs at leats a NVIDIA Pascal™ GPU or better with compute capability 6.0+. I am not going to discuss about rapids.ai in this post, but if you plan to install LightGBM on your GPU, you will soon enough want to play with rapids.ai as well.
Your simplest choice is probably : GTX 1660 Ti which was released in february 2019 and has a compute capability of 7.5
Among older GPUs which have a compute capability of 6+, the prices change quite often but you could make a good deal below.
Nvidia TITAN Xp, GeForce GTX 1080 Ti, GTX 1080, GTX 1070 Ti, GTX 1070, GTX 1050,
But keep in mind that with these cards, the support may be abandoned soon enough…
Test if LightGBM supports GPU
If you can run the following python script:
from lightgbm import LGBMClassifier
from sklearn.datasets import make_moons
model = LGBMClassifier(boosting_type='gbdt', num_leaves=31, max_depth=- 1, learning_rate=0.1, n_estimators=300, device = "gpu")
train, label = make_moons(n_samples=300000, shuffle=True, noise=0.3, random_state=None)
model.fit(train, label)
Without this message:
[LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1
Traceback (most recent call last):
File "seq.py", line 11, in <module>
model.fit(train, label)
File "/home/kerneltrip/anaconda3/lib/python3.7/site-packages/lightgbm/sklearn.py", line 800, in fit
callbacks=callbacks)
File "/home/kerneltrip/anaconda3/lib/python3.7/site-packages/lightgbm/sklearn.py", line 595, in fit
callbacks=callbacks)
File "/home/kerneltrip/anaconda3/lib/python3.7/site-packages/lightgbm/engine.py", line 228, in train
booster = Booster(params=params, train_set=train_set)
File "/home/kerneltrip/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py", line 1666, in __init__
ctypes.byref(self.handle)))
File "/home/kerneltrip/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py", line 47, in _safe_call
raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
lightgbm.basic.LightGBMError: GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1
Then, you do not need this tutorial ;)
Setup guide
Though there is some information here, following the instructions did not do the job for me (hence this detailed guide).
GPU drivers
First, you need to have you drivers set up.
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
You may find your device and the drivers using:
:~$ ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
vendor : NVIDIA Corporation
[...]
model : GM107 [GeForce GTX 745]
driver : nvidia-340 - distro non-free
driver : nvidia-384 - distro non-free
driver : nvidia-410 - third-party non-free recommended
driver : nvidia-396 - third-party non-free
[...]
And then, you can run the following, where 410 replaces the recommended version of the driver:
sudo apt-get update
sudo apt-get install --no-install-recommends nvidia-410
sudo apt-get install --no-install-recommends nvidia-opencl-icd-410 nvidia-opencl-dev opencl-headers
LGBM dependencies
The officials instructions are the following, first the prerequisites:
sudo apt-get install --no-install-recommends git cmake build-essential libboost-dev libboost-system-dev libboost-filesystem-dev
(For some reason, I was still missing Boost elements as we will see later)
Building LGBM for the GPU
Time to download LightGBM
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build ; cd build
Let’s try:
cmake -DUSE_GPU=1 ..
Unfortunately, this does not work.
CMake Error at /usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
Could NOT find OpenCL (missing: OpenCL_LIBRARY OpenCL_INCLUDE_DIR)
The official instructions helped, but :
cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ ..
Still does not work. Now the errors about OpenCL are replaced with others related to Boost :
CMake Error at /usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
Could NOT find Boost (missing: Boost_INCLUDE_DIR filesystem system)
(Required is at least version "1.56.0")
Call Stack (most recent call first):
/usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
/usr/local/share/cmake-3.17/Modules/FindBoost.cmake:2145 (find_package_handle_standard_args)
CMakeLists.txt:121 (find_package)
This might be overkill but:
sudo apt-get install libboost-all-dev
Did the job.
The following additional packages will be installed:
icu-devtools libboost-atomic-dev libboost-atomic1.58-dev libboost-atomic1.58.0
libboost-chrono-dev libboost-chrono1.58-dev libboost-chrono1.58.0 libboost-context-dev
libboost-context1.58-dev libboost-context1.58.0 libboost-coroutine-dev libboost-coroutine1.58-dev
libboost-coroutine1.58.0 libboost-date-time-dev libboost-date-time1.58-dev libboost-dev
[...]
Finally:
cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ ..
Outputs:
/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ ..
-- OpenCL include directory: /usr/local/cuda/include
-- Found Boost: /usr/include (found suitable version "1.58.0", minimum required is "1.56.0") found components: filesystem system
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kerneltrip/Codes/LightGBM/build
And I can run:
make -j$(nproc)
Should look like this :
Scanning dependencies of target lightgbm
Scanning dependencies of target _lightgbm
[ 3%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 3%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 4%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[...]
[ 98%] Built target _lightgbm
[100%] Linking CXX executable ../lightgbm
[100%] Built target lightgbm
Everything was successfully built! Time to set up the python. The repo should look like this on your machine :
~/Codes/LightGBM$ tree -d -L 1
.
├── build
├── compute
├── docker
├── docs
├── examples
├── helpers
├── include
├── pmml
├── python-package
├── R-package
├── src
├── swig
├── tests
└── windows
The official instructions recommend the following operations, but I would not recommend them.
sudo apt-get -y install python-pip
sudo -H pip install setuptools numpy scipy scikit-learn -U
cd python-package/
sudo python setup.py install --precompile
cd ..
install python-pip has the habit of conflicting with the pip that you may have.
Instead, install the missing packages step by step.
In my case, I use conda and I was only missing setuptools.
conda install setuptools
python setup.py install --precompile
Did the job! Now…
from lightgbm import LGBMClassifier
from sklearn.datasets import make_moons
model = LGBMClassifier(boosting_type='gbdt', num_leaves=31, max_depth=- 1, learning_rate=0.1, n_estimators=300, device = "gpu")
train, label = make_moons(n_samples=300000, shuffle=True, noise=0.3, random_state=None)
model.fit(train, label)
Run without any issues ! You can observe the GPU usage with glances[gpu] (this will be fast though)
An error message that did not appear on CPU:
Unfortunately, you may find this error message with the GPU (on some datasets), which you did not have on the CPU :(
raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
lightgbm.basic.LightGBMError: Check failed: (best_split_info.left_count) > (0) at /home/kerneltrip/Codes/LightGBM/src/treelearner/serial_tree_learner.cpp, line 613 .
The most recent information I could get is : https://github.com/microsoft/LightGBM/issues/2742 Apparently, the issue happened on the CPU, then it was fixed, but not on the GPU version. This issue has only been raised some hours ago, let’s hope it will be fixed soon enough.