本文首发于个人博客https://kezunlin.me/post/6b505d27/，欢迎阅读最新内容！
full guide tutorial to install and configure deep learning environments on linux server
Quick Guide
prepare
tools
- MobaXterm (for windows)
- ssh + vscode
for windows:
drop files to MobaXterm to upload to server
usezipformat
commands
view disk
du -d 1 -h
df -h
gpu and cpu usage
watch -n 1 nvidia-smi
top
view files and count
wc -l data.csv
# count how many folders
ls -lR | grep '^d' | wc -l
17
# count how many jpg files
ls -lR | grep '.jpg' | wc -l
1360
# view 10 images
ls train | head
ls test | head
link datasets
# link
ln -s srt dest
ln -s /data_1/kezunlin/datasets/ dl4cv/datasets
scp
scp -r node17:~/dl4cv ~/git/
scp -r node17:~/.keras ~/
tmux for background tasks
tmux new -s notebook
tmux ls
tmux attach -t notebook
tmux detach
wget download
# wget
# continue donwload
wget -c url
# background donwload for large file
wget -b -c url
tail -f wget-log
# kill background wget
pkill -9 wget
tips about training large model
terminal 1:
tmux new -s train
conda activate keras
time python train_alexnet.py
terminal 2:
tmux detach
tmux attach -t train
and then close vscode, otherwise bash training process will exit when we close vscode.
cuda driver and toolkits
see cuda-toolkit for cuda driver version
cudatookit version depends on cuda driver version.
install nvidia-drivers
sudo add-apt-repository ppa:graphics-drivers/ppa
sudp apt-get update
sudo apt-cache search nvidia-*
# nvidia-384
# nvidia-396
sudo apt-get -y install nvidia-418
# test
nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
reboot to test again
https://*.com/questions/43022843/nvidia-nvml-driver-library-version-mismatch
install cuda-toolkit(dirvers)
remove all previous nvidia drivers
sudo apt-get -y pruge nvidia-*
go to here and download cuda_10.1
wget -b -c http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run
sudo sh cuda_10.1.243_418.87.00_linux.run
sudo ./cuda_10.1.243_418.87.00_linux.run
vim .bashrc
# for cuda and cudnn
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
check cuda driver version
> cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 418.87.00 Thu Aug 8 15:35:46 CDT 2019
GCC version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)
>nvidia-smi
Tue Aug 27 17:36:35 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
> nvidia-smi -L
GPU 0: Quadro RTX 8000 (UUID: GPU-acb01c1b-776d-cafb-ea35-430b3580d123)
GPU 1: Quadro RTX 8000 (UUID: GPU-df7f0fb8-1541-c9ce-e0f8-e92bccabf0ef)
GPU 2: Quadro RTX 8000 (UUID: GPU-67024023-20fd-a522-dcda-261063332731)
GPU 3: Quadro RTX 8000 (UUID: GPU-7f9d6a27-01ec-4ae5-0370-f0c356327913)
> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
install conda
./Anaconda3-2019.03-Linux-x86_64.sh
[yes]
[yes]
config channels
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/
conda config --set show_channel_urls yes
install libraries
conclusions:
- py37/keras: conda install -y tensorflow-gpu keras==2.2.5
- py37/torch: conda install -y pytorch torchvision
- py36/mxnet: conda install -y mxnet
keras 2.2.5 was released on 2019/8/23.
Add new Applications: ResNet101, ResNet152, ResNet50V2, ResNet101V2, ResNet152V2.
common libraries
conda install -y scikit-learn scikit-image pandas matplotlib pillow opencv seaborn
pip install imutils progressbar pydot pylint
pip install imutilsto avoid downgrade for tensorflow-gpu
py37
cudatoolkit 10.0.130 0
cudnn 7.6.0 cuda10.0_0
tensorflow-gpu 1.13.1
py36
cudatoolkit anaconda/pkgs/main/linux-64::cudatoolkit-10.1.168-0
cudnn anaconda/pkgs/main/linux-64::cudnn-7.6.0-cuda10.1_0
tensorboard anaconda/pkgs/main/linux-64::tensorboard-1.14.0-py36hf484d3e_0
tensorflow anaconda/pkgs/main/linux-64::tensorflow-1.14.0-gpu_py36h3fb9ad6_0
tensorflow-base anaconda/pkgs/main/linux-64::tensorflow-base-1.14.0-gpu_py36he45bfe2_0
tensorflow-estima~ anaconda/cloud/conda-forge/linux-64::tensorflow-estimator-1.14.0-py36h5ca1d4c_0
tensorflow-gpu anaconda/pkgs/main/linux-64::tensorflow-gpu-1.14.0-h0d30ee6_0
imutils only support 36 and 37.
mxnet only support 35 and 36.
details
# remove py35
conda remove -n py35 --all
conda info --envs
conda create -n py37 python==3.7
conda activate py37
# common libraries
conda install -y scikit-learn pandas pillow opencv
pip install imutils
# imutils
conda search imutils
# py36 and py37
# Name Version Build Channel
imutils 0.5.2 py27_0 anaconda/cloud/conda-forge
imutils 0.5.2 py36_0 anaconda/cloud/conda-forge
imutils 0.5.2 py37_0 anaconda/cloud/conda-forge
# tensorflow-gpu and keras
conda install -y tensorflow-gpu keras
# install pytorch
conda install -y pytorch torchvision
# install mxnet
# method 1: pip
pip search mxnet
mxnet-cu80[mkl]/mxnet-cu90[mkl]/mxnet-cu91[mkl]/mxnet-cu92[mkl]/mxnet-cu100[mkl]/mxnet-cu101[mkl]
# method 2: conda
conda install mxnet
# py35 and py36
TensorFlow Object Detection API
home page: home page
download tensorflow models and rename models-master to tfmodels
vim ~/.bashrc
export PYTHONPATH=/home/kezunlin/dl4cv:/data_1/kezunlin/tfmodels/research:$PYTHONPATH
source ~/.bashrc
jupyter notebook
conda activate py37
conda install -y jupyter
install kernels
python -m ipykernel install --user --name=py37
Installed kernelspec py37 in /home/kezunlin/.local/share/jupyter/kernels/py37
config for server
python -c "import IPython;print(IPython.lib.passwd())"
Enter password:
Verify password:
sha1:ef2fb2aacff2:4ea2998699638e58d10d594664bd87f9c3381c04
jupyter notebook --generate-config
Writing default config to: /home/kezunlin/.jupyter/jupyter_notebook_config.py
vim .jupyter/jupyter_notebook_config.py
c.NotebookApp.ip = '*'
c.NotebookApp.password = u'sha1:xxx:xxx'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
c.NotebookApp.enable_mathjax = True
run jupyter on background
tmux new -s notebook
jupyter notebook
# ctlr+b+d exit session and DO NOT close session
# ctlr+d exit session and close session
access web and input password
test
py37
import cv2
cv2.__version
import tensorflow as tf
import keras
import torch
import torchvision
cat .keras/keras.json
{
"epsilon": 1e-07,
"floatx": "float32",
"backend": "tensorflow",
"image_data_format": "channels_last"
}
py36
import mxnet
train demo
export
# use CPU only
export CUDA_VISIBLE_DEVICES=""
# use gpu 0 1
export CUDA_VISIBLE_DEVICES="0,1"
code
import os
os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
start train
python train.py
./keras folder
view keras models and datasets
ls .keras/
datasets keras.json models
models saved to
/home/kezunlin/.keras/models/
datasets saved to/home/kezunlin/.keras/datasets/
models lists
xxx_kernels_notop.h5forinclude_top = Falsexxx_kernels.h5forinclude_top = True
Datasets
mnist
cifar10
to skip download
wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
mv ~/Download/cifar-10-python.tar.gz ~/.keras/datasets/cifar-10-batches-py.tar.gz
to load data
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
flowers-17
animals
panda images are WRONG !!!
counts
ls -lR animals/cat | grep ".jpg" | wc -l
1000
ls -lR animals/dog | grep ".jpg" | wc -l
1000
ls -lR animals/panda | grep ".jpg" | wc -l
1000
kaggle cats vs dogs
caltech101
download background
wget -b -c http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz
Kaggle API
install and config
see kaggle-api
conda activate keras
conda install kaggle
# download kaggle.json
mv kaggle.json ~/.kaggle/kaggle.json
chmod 600 ~/.kaggle/kaggle.json
cat kaggle.json
{"username":"xxx","key":"yyy"}
or by export
export KAGGLE_USERNAME=xxx
export KAGGLE_KEY=yyy
tips
- go to account and select 'Create API Token' and
keras.jsonwill be downloaded.- Ensure
kaggle.jsonis in the location~/.kaggle/kaggle.jsonto use the API.
check version
kaggle --version
Kaggle API 1.5.5
commands overview
commands
kaggle competitions {list, files, download, submit, submissions, leaderboard}
kaggle datasets {list, files, download, create, version, init}
kaggle kernels {list, init, push, pull, output, status}
kaggle config {view, set, unset}
download datasets
kaggle competitions download -c dogs-vs-cats
show leaderboard
kaggle competitions leaderboard dogs-vs-cats --show
teamId teamName submissionDate score
------ --------------------------------- ------------------- -------
71046 Pierre Sermanet 2014-02-01 21:43:19 0.98533
66623 Maxim Milakov 2014-02-01 18:20:58 0.98293
72059 Owen 2014-02-01 17:04:40 0.97973
74563 Paul Covington 2014-02-01 23:05:20 0.97946
74298 we've been in KAIST 2014-02-01 21:15:30 0.97840
71949 orchid 2014-02-01 23:52:30 0.97733
set default competition
kaggle config set --name competition --value dogs-vs-cats
- competition is now set to: dogs-vs-cats
kaggle config set --name competition --value dogs-vs-cats-redux-kernels-edition
dogs-vs-cats
dogs-vs-cats-redux-kernels-edition
submit
kaggle c submissions
- Using competition: dogs-vs-cats
- No submissions found
kaggle c submit -f ./submission.csv -m "first submit"
competition has already ended, so can not submit.
Nvidia-docker and containers
install
sudo apt-get -y install docker
# Install nvidia-docker2 and reload the Docker daemon configuration
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd
restart (optional)
cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
sudo systemctl enable docker
sudo systemctl start docker
if errors occur:
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
check/etc/docker/daemon.json
test
sudo docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi
sudo nvidia-docker run --rm nvidia/cuda:10.1-base nvidia-smi
Thu Aug 29 00:11:32 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro RTX 8000 Off | 00000000:02:00.0 Off | Off |
| 43% 67C P2 136W / 260W | 46629MiB / 48571MiB | 17% Default |
+-------------------------------+----------------------+----------------------+
| 1 Quadro RTX 8000 Off | 00000000:03:00.0 Off | Off |
| 34% 54C P0 74W / 260W | 0MiB / 48571MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Quadro RTX 8000 Off | 00000000:82:00.0 Off | Off |
| 34% 49C P0 73W / 260W | 0MiB / 48571MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Quadro RTX 8000 Off | 00000000:83:00.0 Off | Off |
| 33% 50C P0 73W / 260W | 0MiB / 48571MiB | 3% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
add user to
dockergroup, and no need to usesudo docker xxx
command refs
sudo nvidia-docker run --rm nvidia/cuda:10.1-base nvidia-smi
sudo nvidia-docker -t -i --privileged nvidia/cuda bash
sudo docker run -it --name kzl -v /home/kezunlin/workspace/:/home/kezunlin/workspace nvidia/cuda
Reference
History
- 20190821: created.
Copyright
- Post author: kezunlin
- Post link: https://kezunlin.me/post/6b505d27/
- Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 3.0 unless stating additionally.