# 07/20/2019 # 我刚想处理这个问题，好了个屁的。这种最可怕了。
问题背景
使用docker push命令推送镜像。
在某个镜像层推送将要结束的时候，收到如下错误信息：
# docker push docker-registry.example.com/project/image-foo:v2.5.0 The push refers to repository [docker-registry.example.com/project/image-foo:v2.5.0] 3d10bce3ce31: Layer already exists 1b70b131f01d: Layer already exists 2c5ef295a217: Pushing [==================================================>] 375.4MB 2ce4f2676a16: Layer already exists 35bbe47ffc77: Layer already exists 05b61dc1031f: Layer already exists d6d37cde236f: Layer already exists ecbc53aebc27: Layer already exists 1585039add0a: Layer already exists 692d855fb28e: Layer already exists 717b092b8c86: Layer already exists received unexpected HTTP status: 504 Gateway Time-out
查看/var/log/registry.log文件，日志中有如下信息（经过了简单的处理）：
# Jul 22 18:26:55 172.18.0.1 proxy[1114]: 2019/07/22 10:26:55 [error] 8#0: *11392 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.50.63, server: , request: "PATCH /v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9 HTTP/1.1", upstream: "http://172.18.0.6:8080/registryproxy/v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9", host: "docker-registry.cluster.dfwsgroup.cn" # Jul 22 18:26:55 172.18.0.1 proxy[1114]: 192.168.50.63 - "PATCH /v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9 HTTP/1.1" 504 176 "-" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.18.12-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \x5C(linux\x5C))" 95.187 95.186 .
环境信息
操作系统：CentOS Linux release 7.4.1708 (Core) 这种问题和操作的关系不大。
软件版本：Harbor v1.4.0
所需资料
仓库地址：https://github.com/goharbor/harbor/tree/v1.4.0
安装配置：Installation & Configuration Guide
用户手册：User Guide
软件架构：Architecture Overview of Harbor
问题排查
暂缓问题
「replication for big images failed with Gateway Time-out intermittently #3446」
修改common/templates/nginx/nginx.http(s).conf文件中的location /v2/部分，添加proxy_send_timeout 900;与proxy_read_timeout 900;配置行。因为从Nginx的日志看是访问/v2/时产生的超时。
可能原因
系统磁盘IO存在问题：
#1 在进行镜像传输时候，系统的「平均负载」比较高，四核，处于3.5以上。而‘‘iostat‘‘命令显示系统IO并不是很高。
因此推测是IO较低，导致镜像写入磁盘的时间比较长，由于超过了Nginx的等待时间，而返回了504错误。
另外，系统中并没有相关Harbor错误日志。在时间点附近，大部分日志都是正常的。
参考文献
docker registry keep showing manifest unknown error for image pull after somedays of push image #17031
Delete tag failed #6515