在隔离环境中搭建 Mirror 站点

2024-02-21 09:15:28

搭建 Mirror 站点的工具有很多，但从根源上讲，他们都使用了 rsync 作为远程同步的工具和手段，因此可称这一类镜像工具为 rsync 镜像工具。在诸多 rsync 镜像工具，使用和配置的便捷性当属 清华大学 开放出的 tunasync 工具，当然它也是一个 Golang 开发而成的 rsync 镜像工具。

之所以会展示这篇操作文档，是因为今天有个我去年带过的学生需要在隔离环境中搭建一个 对阿里云CentOS站点进行镜像的站点，由于操作不当导致镜像失败、且无法Web访问服务。

以下是我使用 tunasync 搭建隔离环境中的 Mirror 站点的过程：

OS环境为 CentOS 7 1804 64位

先对 CentOS 7 做初始化，初始化脚本如下：

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.original
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all && yum makecache
yum -y update
systemctl stop firewalld && systemctl disable firewalld
sed -i 's/^SELINUX=./SELINUX=disabled/' /etc/selinux/config && setenforce 0
wget https://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
yum install -y https://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i 's|^#baseurl=https://download.fedoraproject.org/pub|baseurl=https://mirrors.aliyun.com|' /etc/yum.repos.d/epel
sed -i 's|^metalink|#metalink|' /etc/yum.repos.d/epel*
yum clean all && yum makecache
yum -y update
yum install -y ntpdate
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
yum localinstall -y google-chrome-stable_current_x86_64.rpm
ntpdate cn.ntp.org.cn
yum install -y tree
who | grep googlebigtable | sed -n '1p' | cut -d' ' -f 1 | sort | uniq
DescriptionUser=$(who | grep googlebigtable | sed -n '1p' | cut -d' ' -f 1 | sort | uniq)
echo $DescriptionUser
echo "$DescriptionUser ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
init 6

创建用户mirrors及用户组mirrorgroup
[googlebigtable@localhost tunasyncMirrorWeb]$ sudo su

[root@localhost tunasyncMirrorWeb]# groupadd mirrorgroup

[root@localhost tunasyncMirrorWeb]# useradd -g mirrorgroup mirrors

[root@localhost tunasyncMirrorWeb]# id mirrors

uid=1001(mirrors) gid=1001(mirrorgroup) groups=1001(mirrorgroup)

[root@localhost tunasyncMirrorWeb]# passwd mirrors

Changing password for user mirrors.

New password:

BAD PASSWORD: The password fails the dictionary check - it is based on a dictionary word

Retype new password:

passwd: all authentication tokens updated successfully.

[root@localhost tunasyncMirrorWeb]# id mirrors

uid=1001(mirrors) gid=1001(mirrorgroup) groups=1001(mirrorgroup)

[root@localhost tunasyncMirrorWeb]#

建立应用程序及数据目录
[root@localhost tunasyncMirrorWeb]# mkdir -p /opt/mirrors/tunasync/{conf,db,log}

[root@localhost tunasyncMirrorWeb]# ls -F /opt/mirrors/tunasync/

conf/ db/ log/

[root@localhost tunasyncMirrorWeb]#

[root@localhost tunasyncMirrorWeb]# mkdir -p /opt/mirrors/mirrorsData

[root@localhost tunasyncMirrorWeb]# ls -F /opt/mirrors/

mirrorsData/ tunasync/

[root@localhost tunasyncMirrorWeb]# chown -R mirrors:mirrorgroup /opt/mirrors/mirrorsData

[root@localhost tunasyncMirrorWeb]# ll /opt/mirrors/

total 0

drwxr-xr-x 2 mirrors mirrorgroup 6 Sep 20 06:03 mirrorsData

drwxr-xr-x 4 root root 28 Sep 20 06:00 tunasync

[root@localhost tunasyncMirrorWeb]#

部署 tunasync
[root@localhost tunasyncMirrorWeb]# ls -F

tunasync-linux-amd64-bin.tar.gz

[root@localhost tunasyncMirrorWeb]# chmod 777 tunasync-linux-amd64-bin.tar.gz

[root@localhost tunasyncMirrorWeb]# tar -xzvf tunasync-linux-amd64-bin.tar.gz -C /opt/mirrors/tunasync/

tunasync

tunasynctl

[root@localhost tunasyncMirrorWeb]# ls -F /opt/mirrors/tunasync/

conf/ db/ tunasync tunasynctl

[root@localhost tunasyncMirrorWeb]#

配置环境变量
[root@localhost tunasyncMirrorWeb]# chown -R mirrors:mirrorgroup /opt/mirrors

[root@localhost tunasyncMirrorWeb]# ll /opt/mirrors

total 0

drwxr-xr-x 2 mirrors mirrorgroup 6 Sep 20 06:03 mirrorsData

drwxr-xr-x 4 mirrors mirrorgroup 62 Sep 20 06:07 tunasync

[root@localhost tunasyncMirrorWeb]# ll /opt/mirrors/tunasync/

total 28032

drwxr-xr-x 2 mirrors mirrorgroup 6 Sep 20 06:00 conf

drwxr-xr-x 2 mirrors mirrorgroup 6 Sep 20 06:00 db

-rwxr-xr-x 1 mirrors mirrorgroup 17930410 Sep 11 06:24 tunasync

-rwxr-xr-x 1 mirrors mirrorgroup 10770787 Sep 11 06:24 tunasynctl

[root@localhost tunasyncMirrorWeb]# su - mirrors

[mirrors@localhost ~]$ cat -n ~/.bash_profile

 1 # .bash_profile

 2

 3 # Get the aliases and functions

 4 if [ -f ~/.bashrc ]; then

 5 . ~/.bashrc

 6 fi

 7

 8 # User specific environment and startup programs

 9

10 PATH=$PATH:$HOME/.local/bin:$HOME/bin

11

12 export PATH

[mirrors@localhost ~]$ cp ~/.bash_profile{,.original}

[mirrors@localhost ~]$ echo "TUNAPATH=/home/mirrors/tunasync" >> ~/.bash_profile

[mirrors@localhost ~]$ echo "PATH=$PATH:$TUNAPATH" >> ~/.bash_profile

[mirrors@localhost ~]$ cat -n ~/.bash_profile

 1 # .bash_profile

 2

 3 # Get the aliases and functions

 4 if [ -f ~/.bashrc ]; then

 5 . ~/.bashrc

 6 fi

 7

 8 # User specific environment and startup programs

 9

10 PATH=$PATH:$HOME/.local/bin:$HOME/bin

11

12 export PATH

13 TUNAPATH=/home/mirrors/tunasync

14 PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/mirrors/.local/bin:/home/mirrors/bin:

[mirrors@localhost ~]$ source ~/.bash_profile

[mirrors@localhost ~]$

配置 tunasync 的同步服务
[mirrors@localhost ~]$ ls -F /opt/mirrors/tunasync/

conf/ db/ tunasync tunasynctl

[mirrors@localhost ~]$

[mirrors@localhost ~]$ tee > /opt/mirrors/tunasync/conf/manager.conf << EOF

manager.conf

debug = false

[server]

addr = "127.0.0.1"

port = 14242

ssl_cert = ""

ssl_key = ""

[files]

db_type = "bolt"

db_file = "/opt/mirrors/tunasync/db/manager.db"

ca_cert = ""

EOF

按需创建被同步的镜像配置文件，此处以 CentOS 镜像为例：

[mirrors@localhost ~]$ tee > /opt/mirrors/tunasync/conf/worker-centos.conf << EOF

[global]

name = "centos_worker"

log_dir = "/opt/mirrors/tunasync/log/{{.Name}}"

mirror_dir = "/opt/mirrors/mirrorsData"

concurrent = 10

interval = 1440

[manager]

api_base = "http://localhost:14242"

token = "some_token"

ca_cert = ""

[cgroup]

enable = false

base_path = "/sys/fs/cgroup"

group = "tunasync"

[server]

hostname = "localhost"

listen_addr = "127.0.0.1"

listen_port = 16010

ssl_cert = ""

ssl_key = ""

[[mirrors]]

name = "centos"

provider = "rsync"

upstream = "rsync://mirrors.aliyun.com/centos/"

use_ipv6 = false

EOF

[mirrors@localhost ~]$

mirrors@localhost ~]$ ls -F /opt/mirrors/tunasync/

conf/ db/ log/ tunasync tunasynctl

[mirrors@localhost ~]$ ls -F /opt/mirrors/tunasync/conf/

manager.conf worker-centos.conf

[mirrors@localhost ~]$

启动 tunasync同步程序
启动 tunasync 需先启动 manager，后启动 worker。为了便于监控系统进程情况，建立 /opt/mirrors/tunasync/log/plog/ 目录，所有进程的工作日志在该目录中。

用 mirrors 用户执行：

[mirrors@localhost ~]$ /opt/mirrors/tunasync/tunasync manager --config /opt/mirrors/tunasync/conf/manager.conf >> /opt/mirrors/tunasync/log/plog/manager.log &

[1] 60268

[mirrors@localhost ~]$ /opt/mirrors/tunasync/tunasync worker --config /opt/mirrors/tunasync/conf/worker-centos.conf >> /opt/mirrors/tunasync/log/plog/worker-centos.log &

[2] 60284

[mirrors@localhost ~]$

为内网用户开放 web 访问服务
安装 apache 服务器:

[mirrors@localhost ~]$ su root

Password:

[root@localhost mirrors]# yum -y install httpd

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile

Installed:

httpd.x86_64 0:2.4.6-93.el7.centos

Dependency Installed:

httpd-tools.x86_64 0:2.4.6-93.el7.centos mailcap.noarch 0:2.1.41-2.el7

Complete!

[root@localhost mirrors]#

[root@localhost mirrors]# tree /etc/httpd/

/etc/httpd/

├── conf

│ ├── httpd.conf

│ └── magic

├── conf.d

│ ├── autoindex.conf

│ ├── README

│ ├── userdir.conf

│ └── welcome.conf

├── conf.modules.d

│ ├── 00-base.conf

│ ├── 00-dav.conf

│ ├── 00-lua.conf

│ ├── 00-mpm.conf

│ ├── 00-proxy.conf

│ ├── 00-systemd.conf

│ └── 01-cgi.conf

├── logs -> ../../var/log/httpd

├── modules -> ../../usr/lib64/httpd/modules

└── run -> /run/httpd

6 directories, 13 files

[root@localhost mirrors]# cp /etc/httpd/conf/httpd.conf{,.original}

[root@localhost mirrors]#

[mirrors@localhost ~]$ grep "^\s[^# \t].$" /etc/httpd/conf/httpd.conf | cat

ServerRoot "/etc/httpd"

Listen 80

Include conf.modules.d/*.conf

User apache

Group apache

ServerAdmin root@localhost

<Directory />

AllowOverride none

Require all denied

</Directory>

DocumentRoot "/var/www/html"

<Directory "/var/www">

AllowOverride None

Require all granted

</Directory>

<Directory "/var/www/html">

Options Indexes FollowSymLinks

AllowOverride None

Require all granted

</Directory>

<IfModule dir_module>

DirectoryIndex index.html

</IfModule>

<Files ".ht*">

Require all denied

</Files>

ErrorLog "logs/error_log"

LogLevel warn

<IfModule log_config_module>

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

<IfModule logio_module>

  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

</IfModule>

CustomLog "logs/access_log" combined

</IfModule>

<IfModule alias_module>

ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

<Directory "/var/www/cgi-bin">

AllowOverride None

Options None

Require all granted

</Directory>

<IfModule mime_module>

TypesConfig /etc/mime.types

AddType application/x-compress .Z

AddType application/x-gzip .gz .tgz

AddType text/html .shtml

AddOutputFilter INCLUDES .shtml

</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>

MIMEMagicFile conf/magic

</IfModule>

EnableSendfile on

IncludeOptional conf.d/*.conf

[mirrors@localhost ~]$

修改/etc/httpd/conf/httpd.conf后为：

[mirrors@localhost ~]$ grep "^\s[^# \t].$" /etc/httpd/conf/httpd.conf | cat

ServerRoot "/etc/httpd"

Listen 80

Include conf.modules.d/*.conf

User apache

Group apache

ServerAdmin root@localhost

<Directory />

AllowOverride none

Require all denied

</Directory>

DocumentRoot "/opt/mirrors/mirrorsData/"

<Directory "/var/www">

AllowOverride None

Require all granted

</Directory>

<Directory "/opt/mirrors/mirrorsData/">

Options Indexes FollowSymLinks

AllowOverride None

Require all granted

</Directory>

<IfModule dir_module>

DirectoryIndex index.html

</IfModule>

<Files ".ht*">

Require all denied

</Files>

ErrorLog "logs/error_log"

LogLevel warn

<IfModule log_config_module>

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

<IfModule logio_module>

  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

</IfModule>

CustomLog "logs/access_log" combined

</IfModule>

<IfModule alias_module>

ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"

</IfModule>

<Directory "/var/www/cgi-bin">

AllowOverride None

Options None

Require all granted

</Directory>

<IfModule mime_module>

TypesConfig /etc/mime.types

AddType application/x-compress .Z

AddType application/x-gzip .gz .tgz

AddType text/html .shtml

AddOutputFilter INCLUDES .shtml

</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>

MIMEMagicFile conf/magic

</IfModule>

EnableSendfile on

IncludeOptional conf.d/*.conf

[mirrors@localhost ~]$

[mirrors@localhost ~]$ chmod -R 755 /opt/mirrors/mirrorsData

[mirrors@localhost ~]$ systemctl start httpd.service

[mirrors@localhost ~]$ systemctl enable httpd.service

Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

[mirrors@localhost ~]$

[root@localhost Public]# chown -R mirrors:mirrorgroup /opt/mirrors/mirrorsData/index.html

[mirrors@localhost ~]$ cat /opt/mirrors/mirrorsData/index.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">;

<html xmlns="http://www.w3.org/1999/xhtml">;

<head>

<meta http-equiv="content-type" content="text/html; charset=utf-8"/>

<link rel="stylesheet" type="text/css" href="/.resource/css/mirror.css" media="screen" />

<script type="text/javascript" src=".resource/js/jquery-1.11.3.min.js" ></script>

<script type="text/javascript" src=".resource/js/mirror.js" ></script>

<script type="text/javascript" src=".resource/js/main.js" ></script>

<title>CentOS</title>

</head>

<body>

<h1>欢迎访问CentOS镜像站</h1>

<table id="distro-table" cellpadding="0" cellspacing="0">

<colgroup>

    <col width="12%"/>

    <col width="15%"/>

    <col width="35%"/>

    <col width="10%"/>

    <col width="8%"/>

    <col width="20%"/>

</colgroup>

<thead>

    <tr>

        <th>镜像名</th>

        <th>上次更新时间</th>

        <th>更新源</th>

        <th>状态</th>

        <th>镜像大小</th>

        <th>使用帮助</th>

    </tr>

</thead>

<tbody>

    <tr class="odd">

        <td><a href="/centos/">centos/</a></td>

        <td class="centos update-time">N/A</td>

        <td class="centos upstream">N/A</td>

        <td class="centos sync-status">N/A</td>

        <td class="centos mirror-size">N/A</td>

        <td><a href="/.help/centos.html">Centos/Redhat 使用帮助</a></td>

    </tr>

</tbody>

</table>

<div id="footer">

</div>

</body>

</html>

[mirrors@localhost ~]$

[mirrors@localhost ~]$ chmod -R 755 /opt/mirrors/mirrorsData

至此 Mirror 站点搭建完成，可以通过浏览器直接访问 站点 IP 访问 ：

在隔离环境中搭建 Mirror 站点

在隔离环境中搭建 Mirror 站点

上一篇：一些docker操作


下一篇：CentOS系列教程（九）