0. Operating System and Distribution
The article is using ubuntu 16.4 (xenial) as an example. Other distributions can be setup similarly.
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"
Software versions used here are:
- docker: 18.09.1
- kubernetes: v1.13.2
- containerd: v1.2.2
- Kata Containers: 1.5.0
1. Install docker
- Install docker following the official document
- Optionally configure normal users to run docker commandline
2. Install Kata Containers
$ ARCH=$(arch)
$ sudo sh -c "echo ‘deb http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/master/xUbuntu_$(lsb_release -rs)/ /‘ > /etc/apt/sources.list.d/kata-containers.list"
$ curl -sL http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/master/xUbuntu_$(lsb_release -rs)/Release.key | sudo apt-key add -
$ sudo apt update
$ sudo apt install -y kata-runtime kata-proxy kata-shim
Other installation methods can be found in the official document.
3. Configure docker to use Kata Containers
3.1 Configure docker Daemon
Create /etc/docker/daemon.json file if not existing. Make sure following configuration is in the file:
{
"registry-mirrors": ["https://registry.docker-cn.com"],
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"default-runtime": "runc",
"runtimes": {
"kata-runtime": {
"path": "kata-runtime"
}
}
}
3.2 Restart docker Daemon
sudo systemctl restart docker
3.3 Verify docker with Kata Containers
$ uname -r
4.4.0-57-generic
$ docker run --rm -it --runtime kata-runtime busybox uname -r
4.14.67-16.container
Here container kernel version is different than the one on the host.
4. Install and Configure containerd
4.1 Install containerd
$ wget https://storage.googleapis.com/cri-containerd-release/cri-containerd-cni-1.2.2.linux-amd64.tar.gz
$ sudo tar --no-overwrite-dir -C / -xzf cri-containerd-cni-1.2.2.linux-amd64.tar.gz
4.2 Configure containerd CNI Plugin
$ sudo mkdir -p /etc/cni/net.d
$ sudo bash -c ‘cat >/etc/cni/net.d/10-containerd-net.conflist <<EOF
{
"cniVersion": "0.3.1",
"name": "containerd-net",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"promiscMode": true,
"ipam": {
"type": "host-local",
"subnet": "10.88.0.0/16",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true}
}
]
}
EOF‘
4.3 Configure containerd
Create /etc/containerd/config.toml file with following contents:
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0
[grpc]
address = "/run/containerd/containerd.sock"
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[debug]
address = ""
uid = 0
gid = 0
level = "debug"
[metrics]
address = ""
grpc_histogram = false
[cgroup]
path = ""
[plugins]
[plugins.cgroups]
no_prometheus = false
[plugins.cri]
stream_server_address = "127.0.0.1"
stream_server_port = "0"
enable_selinux = false
sandbox_image = "k8s.gcr.io/pause:3.1"
stats_collect_period = 10
systemd_cgroup = false
enable_tls_streaming = false
max_container_log_line_size = 16384
[plugins.cri.containerd]
snapshotter = "overlayfs"
no_pivot = false
[plugins.cri.containerd.runtimes]
[plugins.cri.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v1"
[plugins.cri.containerd.runtimes.runc.options]
NoPivotRoot = false
NoNewKeyring = false
ShimCgroup = ""
IoUid = 0
IoGid = 0
BinaryName = "/usr/local/sbin/runc"
Root = ""
CriuPath = ""
SystemdCgroup = false
[plugins.cri.containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
[plugins.cri.containerd.runtimes.kata.options]
[plugins.cri.cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
conf_template = ""
[plugins.cri.x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
[plugins.diff-service]
default = ["walking"]
[plugins.linux]
shim = "containerd-shim"
runtime = "runc"
runtime_root = ""
no_shim = false
shim_debug = false
[plugins.opt]
path = "/opt/containerd"
[plugins.restart]
interval = "10s"
[plugins.scheduler]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"
4.4 Configure containerd to use proxy (optional)
If the host is behind a proxy, modify /lib/systemd/system/containerd.service to add Environment option under [Service] section, like:
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target
[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
Environment="HTTP_PROXY=192.168.80.1:12343"
[Install]
WantedBy=multi-user.target
3.5 Restart containerd daemon
$ sudo systemctl daemon-reload
$ sudo systemctl restart containerd
3.6 Verify containerd with Kata Containers
Create two testing yaml files:
$ cat > pod.yaml << EOF
metadata:
attempt: 1
name: busybox-sandbox
namespace: default
uid: hdishd83djaidwnduwk28bcsb
log_directory: /tmp
linux:
namespaces:
options: {}
EOF
$ cat > container.yaml << EOF
metadata:
name: busybox
image:
image: busybox:latest
command:
- top
log_path: busybox.0.log
EOF
Test containerd with crictl:
$ sudo crictl pull busybox
$ sudo crictl pull k8s.gcr.io/pause:3.1
$ sudo crictl runp -r kata pod.yaml
63f3f0d050745f1c48cfac24045de4cf01c801c48e5b850b73048f9330f533d2
$ sudo crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
63f3f0d050745 2 minutes ago Ready busybox-sandbox default 1
$ sudo crictl create 63f3f0d050745 container.yaml pod.yaml
d6d4169f06c2cdce479f734fa5d6db9fedb95a7c47b202dc1cc0376f06cfbfe1
$ sudo crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID
d6d4169f06c2c busybox:latest 28 seconds ago Created busybox 0 63f3f0d050745
$ sudo crictl start d6d4169f06c2c
d6d4169f06c2c
$ sudo crictl exec -it d6d4169f06c2c uname -r
4.14.67-16.container
$ uname -r
4.4.0-57-generic
Here container kernel version is different than the one on the host.
5. Install and Configure Kubernetes
5.1 Prepare Nodes
On all Kubernetes nodes, repeat above 1-4 steps to install and configure docker, containerd and Kata Containers.
5.2 Install kubeadm,kubelet,kubectl
Following the Kubernetes official document to install kubeadm, kubelet and kubectl on all nodes.
5.3 Create kubeadm Config
On Kubernetes master node, create a kubeadm config file:
$ cat > config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.121.15
bindPort: 6443
nodeRegistration:
criSocket: /run/containerd/containerd.sock
name: ubuntu1604.ubuntu1604
kubeletExtraArgs:
"feature-gates": "RuntimeClass=true"
"fail-swap-on": "false"
taints: []
---
apiServer:
timeoutForControlPlane: 4m0s
extraArgs:
"feature-gates": "RuntimeClass=true"
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.13.1
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
EOF
NOTE: modify advertiseAddress to specify the master node IP in above config.yaml file.
5.4 Install kubernetes
$ sudo kubeadm init --config=config.yaml --ignore-preflight-errors=ALL
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.121.86:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:e9909ea05b0045ea99c017e06a6a19d8d4da0a2dbbb11784e9546d5cc061ab70
$ mkdir -p $HOME/.kube
$ sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
5.5 Install CNI for Kubernetes
Using flannel as an example. Other CNI plugins can be installed per Kubernetes document
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
Wait a few minutes, see if all control plane pods are up and running:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-86c58d9df4-qbgs2 1/1 Running 0 4m29s
kube-system coredns-86c58d9df4-rkgvd 1/1 Running 0 4m29s
kube-system etcd-ubuntu1604.ubuntu1604 1/1 Running 0 3m39s
kube-system kube-apiserver-ubuntu1604.ubuntu1604 1/1 Running 0 3m31s
kube-system kube-controller-manager-ubuntu1604.ubuntu1604 1/1 Running 0 3m25s
kube-system kube-flannel-ds-amd64-ccv2g 1/1 Running 0 93s
kube-system kube-proxy-nkp8t 1/1 Running 0 4m29s
kube-system kube-scheduler-ubuntu1604.ubuntu1604 1/1 Running 0 3m50s
5.6 Setup Node Access Permission on Cluster Resources
$ kubectl set subject clusterrolebinding system:node --group=system:nodes
5.7 Configure Kubernetes Slave Nodes
Based on the config file generated by kubeadm config print join-defaults, create a customised config file for slave nodes, changing apiServerEndpoint to the IP of Kubernetes master.
$ cat > cluster.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta1
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
bootstrapToken:
apiServerEndpoint: 192.168.121.86:6443
token: abcdef.0123456789abcdef
unsafeSkipCAVerification: true
timeout: 5m0s
tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
criSocket: /run/containerd/containerd.sock
name: k8s-kata
kubeletExtraArgs:
"feature-gates": "RuntimeClass=true"
"fail-swap-on": "false"
EOF
Then, to join a node in Kubernetes cluster, run the following command:
$ sudo kubeadm join --config cluster.yaml --ignore-preflight-errors=ALL
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run ‘kubectl get nodes‘ on the master to see this node join the cluster.
Wait a moment, see if the new node is in Ready status:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-kata Ready <none> 19m v1.13.2
ubuntu1604.ubuntu1604 Ready master 108m v1.13.2
6. Configure Kubernetes RuntimeClass
6.1 Create Kata Containers Runtime Class
$ cat > kata_resource.yaml << EOF
apiVersion: node.k8s.io/v1beta1 # RuntimeClass is defined in the node.k8s.io API group
kind: RuntimeClass
metadata:
name: kataclass # The name the RuntimeClass will be referenced by
# RuntimeClass is a non-namespaced resource
handler: kata
EOF
$ kubectl apply -f kata_resource.yaml
runtimeclass.node.k8s.io/kataclass created
6.2 Verify that Kata Containers Runtime Class is created:
$ kubectl get runtimeclasses
NAME RUNTIME-HANDLER AGE
kataclass kata 49s
6.3 Test Kata Containers Via Runtime Class
In a pod spec, set runtimeClassName as kataclass to ask Kubernetes to use Kata Containers:
$ cat > pod-kata.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
name: foobar-kata
spec:
runtimeClassName: kataclass
containers:
- name: nginx
image: nginx
EOF
$ kubectl apply -f pod-kata.yaml
pod/foobar-kata created
Wait a bit and verify that if pod is successfully created:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
foobar-kata 0/1 ContainerCreating 0 5s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
foobar-kata 1/1 Running 0 7s
See if pod kernel version is different than the host one:
$ kubectl exec -it foobar-kata bash
root@foobar-kata:/# uname -r
4.14.67-4.container
root@foobar-kata:/# exit
exit
$ uname -r
4.4.0-57-generic
Repeatedly join all nodes in your cluster to the Kubernetes cluster. Then voilà, congrats! A Kubernetes cluster is up and running with containerd and Kata Containers.