• 欢迎订阅我的Youtube频道
  • 欢迎加入我的电报群

How to create a Machine Learning VM from scratch

网络技术 fennng 4个月前 (10-30) 226次浏览 0个评论

The quickest way to own a Azure Deep Learning VM with CUDA GPU support.

Pre-requisites: You will need to have a azure subscription

Firstly, create a NC6 VM with Ubuntu 16.4 LTS, make sure HDD is selected and port 80, 443 and 22 are open. This VM comes with a Tesla K80 GPU.

In my case, I have to select East US before I could find NC6 in the list.

SSH to the VM

Install docker

Don’t use the following command to install docker, as it’s an old version: Docker version 18.09.7, build 2d0083d

#don't run this
sudo apt install docker.io

Follow this page to install the latest docker

https://docs.docker.com/install/linux/docker-ce/ubuntu/

sudo apt-get install     apt-transport-https     ca-certificates     curl     gnupg-agent     software-properties-common
sudo apt-key fingerprint 0EBFCD88
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
docker -v  
Docker version 19.03.4, build 9013bf583a

Then Install Nvidia Driver

The info from this page is useful, but don’t build the gpu docker using this docker file.
https://hub.docker.com/r/floydhub/dl-docker

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update 
sudo apt-get install nvidia-352

Restart VM with following command

sudo shutdown -r now

check driver

cat /proc/driver/nvidia/version
[email protected]:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  384.130  Wed Mar 21 03:37:26 PDT 2018
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)

install nvidia docker

# Add the package repositories
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

Test the enviroment

docker run --gpus all nvidia/cuda:9.0-base nvidia-smi


Don’t do this — Build the docker, this will take a while, about 20 minutes

#don't run this, as the built image doesn't work properly.
git clone https://github.com/saiprashanths/dl-docker.git
cd dl-docker
docker build -t floydhub/dl-docker:gpu -f Dockerfile.gpu .

The reason of the failure as shown below.

So I switch to the following docker image.

https://hub.docker.com/r/jihong/keras-gpu/

sudo docker run --gpus all -it --rm -p 8888:8888 -v /sharefolder:/root/workspace jihong/keras-gpu bash
jupyter notebook --allow-root

Now, the jupyter notebook is ready. We need to find a easy way to access it. So I use ngrok here. For proper setup, nginx-proxy with SSL is a better option.

Install ngrok following this instruction
https://dashboard.ngrok.com/get-started

#in my case
wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
sudo apt install unzip  
unzip ngrok-stable-linux-amd64.zip
mv ngrok /usr/bin/ngrok

Run ngrok

ngrok http 8888

Copy the jupyter notebook token from the linux bash to login

Then navigate to /sharefolder and run

git clone https://github.com/fengnz/deep-learning-with-python-notebooks.git

Then you will be able to see a bunch of notebook to start with

Pickup the first one and have a play.


峰享技术 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:How to create a Machine Learning VM from scratch
喜欢 (0)
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址