kubernetes（十二）部署 harbor 私有仓库

2024-01-07 19:46:40

安装下载依赖包

安装docker-compose

从 docker compose 发布页面下载最新的 docker-compose 二进制文件,本文以1.25.4为例

cd /opt/k8s/work/harbor/

wget https://github.com/docker/compose/releases/download/1.25.4/docker-compose-Linux-x86_64

mv docker-compose-Linux-x86_64 /opt/k8s/bin/docker-compose
chmod a+x  /opt/k8s/bin/docker-compose
export PATH=/opt/k8s/bin:$PATH

下载harbor二进制文件

从 harbor 发布页面下载最新的 harbor 离线安装包，本文以v1.9.4为例

cd /opt/k8s/work/harbor/

wget https://github.com/goharbor/harbor/releases/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz

tar -xzvf harbor-offline-installer-v1.5.1.tgz

导入 docker images
导入离线安装包中 harbor 相关的 docker images
```
cd /opt/k8s/work/harbor/harbor
docker load -i harbor.v1.9.4.tar.gz
```

以http形式启动

修改harbor.yml文件
修改 hostname、data_volume属性值

cd /opt/k8s/work/harbor/harbor

cp harbor.yml harbor.yml.bak

vim harbor.yml

修改信息如下

diff harbor.yml harbor.yml.bak
5c5
< hostname: 192.168.0.107
---
> hostname: reg.mydomain.com
40c40
< data_volume: /data/k8s/harbor/data
---
> data_volume: /data

加载和启动 harbor 镜像

cd /opt/k8s/work/harbor/harbor
mkdir -p /data/k8s/harbor/data
chmod 777 /var/run/docker.sock /data/k8s/harbor/data
./install.sh

启动日志


Note: docker version: 18.09.6

Note: docker-compose version: 1.25.4
# Configuration file of Harbor

[Step 1]: loading Harbor images ...
Loaded image: goharbor/harbor-core:v1.9.4
Loaded image: goharbor/clair-photon:v2.1.0-v1.9.4
Loaded image: goharbor/harbor-portal:v1.9.4
Loaded image: goharbor/nginx-photon:v1.9.4
Loaded image: goharbor/chartmuseum-photon:v0.9.0-v1.9.4
Loaded image: goharbor/prepare:v1.9.4
Loaded image: goharbor/redis-photon:v1.9.4
Loaded image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.4
Loaded image: goharbor/notary-server-photon:v0.6.1-v1.9.4
Loaded image: goharbor/harbor-log:v1.9.4
Loaded image: goharbor/harbor-db:v1.9.4
Loaded image: goharbor/harbor-jobservice:v1.9.4
Loaded image: goharbor/harbor-registryctl:v1.9.4
Loaded image: goharbor/notary-signer-photon:v0.6.1-v1.9.4
Loaded image: goharbor/harbor-migrator:v1.9.4


[Step 2]: preparing environment ...
prepare base dir is set to /opt/k8s/work/harbor/harbor
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 3]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db     ... done
Creating redis         ... done
Creating harbor-portal ... done
Creating registry      ... done
Creating registryctl   ... done
Creating harbor-core   ... done
Creating harbor-jobservice ... done
Creating nginx             ... done

✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at http://192.168.0.107.
For more details, please visit https://github.com/goharbor/harbor .

查看启动状态

root@master:/opt/k8s/work/harbor/harbor# docker-compose ps
  	Name                     Command                  State                 Ports
---------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
registryctl         /harbor/start.sh                 Up (healthy)

如果有不是healthy状态的对象，到/var/log/harbor/目录下查看对应对象的日志
harbor官方提供的启动文件中容器的日志类型是syslog，不支持用docker logs查看
```
ls /var/log/harbor/
```

core.log jobservice.log portal.log postgresql.log proxy.log redis.log registryctl.log registry.log

	```

浏览器访问 http://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。
![](https://www.icode9.com/i/l/?n=18&i=blog/1918821/202002/1918821-20200214210112961-1493493575.png)

创建一个新的项目

docker命令拉取和上传镜像

执行login

root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://192.168.0.107/v2/: dial tcp 192.168.0.107:443: connect: connection refused

因为docker命令默认采用https和API交互，而我们的harbor是http的，所以不能执行，需要在/etc/docker/daemon.json中追加insecure-registries配置

{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
    "insecure-registries": ["192.168.0.107"],
    "max-concurrent-downloads": 20,
    "live-restore": true,
    "max-concurrent-uploads": 10,
    "data-root": "/data/k8s/docker/data",
    "log-opts": {
      "max-size": "100m",
      "max-file": "5"
    }
}

重新启动docker服务

root@slave:~# systemctl restart docker

root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

向harbor中刚创建的私有项目里面上传镜像

root@slave:~# docker tag k8s.gcr.io/pause:3.1 192.168.0.107/k8s/pause:3.1
root@slave:~# docker push 192.168.0.107/k8s/pause:3.1
The push refers to repository [192.168.0.107/k8s/pause]
e17133b79956: Pushed
3.1: digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d size: 527

浏览器查看

从harbor中下载镜像

root@slave:~# docker rmi  192.168.0.107/k8s/pause:3.1
Untagged: 192.168.0.107/k8s/pause:3.1
Untagged: 192.168.0.107/k8s/pause@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
root@slave:~# docker pull  192.168.0.107/k8s/pause:3.1
3.1: Pulling from k8s/pause
Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
Status: Downloaded newer image for 192.168.0.107/k8s/pause:3.1

以https形式启动

下列操作的工作目录/opt/k8s/work/harbor/harbor是解压harbor离线安装文件后生成的 harbor 目录

停止harbor（如果没有启动过，则跳过该步骤）

cd /opt/k8s/work/harbor/harbor

root@master:/opt/k8s/work/harbor/harbor# COMPOSE_HTTP_TIMEOUT=200 docker-compose down -v
Stopping nginx             ... done
Stopping harbor-jobservice ... done
Stopping harbor-core       ... done
Stopping registryctl       ... done
Stopping redis             ... done
Stopping harbor-portal     ... done
Stopping harbor-db         ... done
Stopping registry          ... done
Stopping harbor-log        ... done
Removing harbor-log        ... done
Removing network harbor_harbor

执行docker-compose down -v后，有可能对应的容器还在，可以执行强制停掉容器
```
docker ps | grep harbor | awk '{print $1}' | xargs -I {} docker rm -f {}
```

创建 harbor nginx 服务器使用的 x509 证书，本文档采用cfssl工具生成，也可以利用openssl生成，具体步骤参考harbor-openssl

创建harbor用的CA根证书

创建配置文件

cd /opt/k8s/work/harbor/
cat > harbor-ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "harbor": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

创建证书签名请求文件

cd /opt/k8s/work/harbor/

cat > harbor-ca-csr.json <<EOF
{
  "CN": "harbor-ca",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "NanJing",
      "L": "NanJing",
      "O": "k8s",
      "OU": "system"
    }
  ],
  "ca": {
    "expiry": "87600h"
 }
}
EOF

生成自签名的根证书,分发证书

cd /opt/k8s/work/harbor/

cfssl gencert -initca harbor-ca-csr.json | cfssljson -bare harbor-ca

ls harbor-ca*pem

mkdir -p /etc/harbor/cert

mv harbor-ca*pem harbor-ca-config.json  /etc/harbor/cert

创建harbor用的证书

生成证书请求文件

cd /opt/k8s/work/harbor/

cat > harbor-server-csr.json <<EOF
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "192.168.0.107"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "NanJing",
      "L": "NanJing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF

hosts 字段指定授权使用该证书的当前部署节点 IP

生成证书


cfssl gencert -ca=/etc/harbor/cert/harbor-ca.pem \

-ca-key=/etc/harbor/cert/harbor-ca-key.pem
-config=/etc/harbor/cert/harbor-ca-config.json
-profile=harbor harbor-server-csr.json | cfssljson -bare harbor-server

		ls harbor-server*pem
		
		mv harbor-server*pem /etc/harbor/cert
		
		```

编辑配置文件

修改 hostname、data_volume属性值,去除默认的http方式，追加https的配置

cd /opt/k8s/work/harbor/harbor

vim harbor.yml

修改内容如下

diff harbor.yml harbor.yml.bak
5c5
< hostname: 192.168.0.107
---
> hostname: reg.mydomain.com
8c8
< #http:
---
> http:
10c10
<   #  port: 80
---
>   port: 80
13c13
< https:
---
> # https:
15c15
<    port: 443
---
> #   port: 443
17,18c17,18
<    certificate: /etc/harbor/cert/harbor-server.pem
<    private_key: /etc/harbor/cert/harbor-server-key.pem
---
> #   certificate: /your/certificate/path
> #   private_key: /your/private/key/path
40c40
< data_volume: /data/k8s/harbor/data
---
> data_volume: /data

生成配置文件（如果没有启动过，则跳过该步骤）

cd /opt/k8s/work/harbor/harbor

./prepare

执行日志

prepare base dir is set to /opt/k8s/work/harbor/harbor
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/core/env
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

修改生成文件的访问权限

cd /opt/k8s/work/harbor/harbor
chmod -R 777 common

启动harbor

如果是首次启动执行

cd /opt/k8s/work/harbor/harbor

mkdir -p /data/k8s/harbor/data
chmod 777 /var/run/docker.sock /data/k8s/harbor/data
./install.sh

否则执行

docker-compose up -d

查看启动状态

root@master:/opt/k8s/work/harbor/harbor# docker-compose ps
  Name                     Command                  State                          Ports
---------------------------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
registryctl         /harbor/start.sh                 Up (healthy)

nginx 暴露了两个端口http的80和https的443，使用http访问80会自动重定向到https

浏览器访问 https://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。
- 其中的k8s是在http模式下创建的项目，直接启动https的可参照上面创建
docker命令拉取和上传镜像

如果之前已经执行过docker login，配置修改后需要重新docker login，需要删除 /root/.docker/config.json文件

如果在/etc/docker/daemon.json中配置过insecure-registries，需要去掉,并重新启动docker服务
1. 执行login
```
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
```

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://192.168.0.107/v2/: x509: certificate signed by unknown authority

	```
	错误是说我们用的证书是自签名的证书，签发证书机构未经认证
	解决方法是将签署 harbor-server 证书的 CA 证书拷贝到 /etc/docker/certs.d/192.168.0.107 目录下（需要在所有要访问harbor的节点上都执行此操作）
	
	```
	root@slave:~# mkdir -p /etc/docker/certs.d/192.168.0.107
	root@slave:~# scp root@192.168.0.107:/etc/harbor/cert/harbor-ca.pem /etc/docker/certs.d/192.168.0.107/ca.crt
	root@192.168.0.107's password:
	harbor-ca.pem                        100% 1306   283.7KB/s   00:00
	root@slave:~# ls /etc/docker/certs.d/192.168.0.107/
	ca.crt
	
	```
	
	重新执行login
	
	```
	root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
	WARNING! Using --password via the CLI is insecure. Use --password-stdin.
	WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
	Configure a credential helper to remove this warning. See
	https://docs.docker.com/engine/reference/commandline/login/#credentials-store
	
	Login Succeeded
	
	```

上传镜像


root@slave:~# docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
kubernetesui/metrics-scraper   v1.0.3              3327f0dbcb4a        2 weeks ago         40.1MB
busybox                        latest              6d5fcfe5ff17        7 weeks ago         1.22MB
coredns/coredns                1.6.6               cc4d8e8c6169        2 months ago        40.8MB
k8s.gcr.io/pause               3.1                 da86e6ba6ca1        2 years ago         742kB
192.168.0.107/k8s/pause        3.1                 da86e6ba6ca1        2 years ago         742kB
kubeimage/pause                3.1                 da86e6ba6ca1        2 years ago         742kB
nginx                          1.9.1               94ec7e53edfc        4 years ago         133MB
root@slave:~# docker tag busybox:latest 192.168.0.107/k8s/busybox:latest
root@slave:~# docker push 192.168.0.107/k8s/busybox:latest
The push refers to repository [192.168.0.107/k8s/busybox]
195be5f8be1d: Pushed
latest: digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 size: 527

通过浏览器查看刚上传的镜像

从harbor下载镜像

root@slave:~# docker rmi 192.168.0.107/k8s/busybox:latest
Untagged: 192.168.0.107/k8s/busybox:latest
Untagged: 192.168.0.107/k8s/busybox@sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
root@slave:~# docker pull 192.168.0.107/k8s/busybox:latest
latest: Pulling from k8s/busybox
Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
Status: Downloaded newer image for 192.168.0.107/k8s/busybox:latest

kubernetes集群中使用私有的harbor镜像仓库

kubelet配置

kubernetes以pod为管理单元，而不是docker容器，在创建pod时，会先启动一个名为k8s.gcr.io/pause:3.1的镜像。

该镜像存在于google的镜像仓库中，国内不能直接访问，需要*下载，如果集群新加节点，或者节点上这个镜像被清理掉，将导致pod不能启动

一般做法是搭建私有镜像仓库，把这个镜像push到我们的私有镜像仓库，再配置kubelet使用我们私有仓库中的pause镜像

具体做法是修改kubelet服务的启动文件:/etc/systemd/system/kubelet.service,在其中追加--pod-infra-container-image属性，属性值指向自己私有仓库中的pause镜像

例如
```
--pod-infra-container-image=192.168.0.107/k8s/pause:3.1
```
重启kubelet服务
```
systemctl restart kubelet
```

kubernetes和harbor进行认证

kubernetes提供了各种方式来访问各种镜像仓库，具体参考Using a Private Registry,本文采用配置节点的方式来使用私有仓库

在集群的任意一个节点上登陆harbor

root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

成功后会生成认证文件/root/.docker/config.json


root@slave:~# ls /root/.docker/
config.json
root@slave:~# cat /root/.docker/config.json
{
        "auths": {
                "192.168.0.107": {
                        "auth": "YWRtaW46SGFyYm9yMTIzNDU="
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/18.09.6 (linux)"
        }
}

配置kubelet使用此认证文件

默认情况下，kubelet会从如下路径中查找私有仓库的认证文件
- {--root-dir:-/var/lib/kubelet}/config.json
- {cwd of kubelet}/config.json
- ${HOME}/.docker/config.json #须在环境变量文件中为 kubelet 显式设置 HOME=/root
- /.docker/config.json
此处我们将私有仓库的认证文件copy到kubelet的root目录下，此目录在kubelet.service中进行设置
例如
```
--root-dir=/data/k8s/k8s/kubelet
```

copy私有仓库的认证文件到kubelet的root目录（所有运行kubelet，需要访问harbor的节点都需要），本环境中有两个节点

cp /root/.docker/config.json /data/k8s/k8s/kubelet/config.json

scp /root/.docker/config.json root@192.168.0.114:/data/k8s/k8s/kubelet/config.json

验证

创建一个pod,image 指定私有仓库里面的镜像

cd /opt/k8s/yml

cat > private-busybox.yml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: private-busybox
spec:
  containers:
  - name: private-busybox
    image: 192.168.0.107/k8s/busybox:latest
    command:
      - sleep
      - "3600"
EOF

创建pod

root@master:/opt/k8s/yml# kubectl create -f private-busybox.yml

查看运行状态

root@master:/opt/k8s/yml# kubectl get pod | grep private-busybox
private-busybox                     1/1     Running   0          108s

通过docker命令，查看启动pod对应的容器的镜像名称

root@master:/opt/k8s/yml# docker ps | grep -e IMAGE -e private-busybox
CONTAINER ID        IMAGE                                                    COMMAND                  CREATED              STATUS                  PORTS                                                           NAMES
c2b43d9d9b3e        192.168.0.107/k8s/busybox                                "sleep 3600"             About a minute ago   Up About a minute                                                                       k8s_private-busybox_private-busybox_default_98a4d4bd-fd46-4d9d-8ef9-95845ce7e53d_0
a4a84461d5fe        192.168.0.107/k8s/pause:3.1                              "/pause"                 About a minute ago   Up About a minute                                                                       k8s_POD_private-busybox_default_98a4d4bd-fd46-4d9d-8ef9-95845ce7e53d_0                   1/1     Running   0          5s

可以看到启动的容器对应的镜像都是我们自己的私有仓库中的镜像

码农公寓

安装下载依赖包

以http形式启动

以https形式启动

kubernetes集群中使用私有的harbor镜像仓库

相关文章