一：各模块属性
| 模块名称 | 状态 | 建议实例数 | 功能 | 负载均衡组件 |
| TiDB | 无状态 | 2 | 接收SQL请求，处理SQL相关逻辑，并通过PB找到存储数据的TiKV地址 | LVS、HAProxy、F5 |
| PB | 集群 | 3 奇数个节点,推荐>3 | 整个集群的管理模块，存储元信息、对TiKV集群进行调度和负载均衡、分配全局事务ID | Raft |
| TiKV | 集群 | 3 | 负责存储数据 | Raft |
二：环境要求
2.1：开发测试环境：
| 组件 | CPU | 内存 | 本地存储 | 网络 | 实例数量(最低要求) |
|---|---|---|---|---|---|
| TiDB | 8核+ | 16 GB+ | 无特殊要求 | 千兆网卡 | 1（可与 PD 同机器） |
| PD | 4核+ | 8 GB+ | SAS, 200 GB+ | 千兆网卡 | 1（可与 TiDB 同机器） |
| TiKV | 8核+ | 32 GB+ | SSD, 200 GB+ | 千兆网卡 | 3 |
2.2：生产环境：
| 组件 | CPU | 内存 | 硬盘类型 | 网络 | 实例数量(最低要求) |
|---|---|---|---|---|---|
| TiDB | 16核+ | 32 GB+ | SAS | 万兆网卡（2块最佳） | 2 |
| PD | 4核+ | 8 GB+ | SSD | 万兆网卡（2块最佳） | 3 |
| TiKV | 16核+ | 32 GB+ | SSD | 万兆网卡（2块最佳） | 3 |
| 监控 | 8核+ | 16 GB+ | SAS | 千兆网卡 | 1 |
2.3：端口说明：
| 组件 | 默认端口 | 说明 |
|---|---|---|
| TiDB | 4000 | 应用及 DBA 工具访问通信端口 |
| TiDB | 10080 | TiDB 状态信息上报通信端口 |
| TiKV | 20160 | TiKV 通信端口 |
| PD | 2379 | 提供 TiDB 和 PD 通信端口 |
| PD | 2380 | PD 集群节点间通信端口 |
| Pump | 8250 | Pump 通信端口 |
| Drainer | 8249 | Drainer 通信端口 |
| Prometheus | 9090 | Prometheus 服务通信端口 |
| Pushgateway | 9091 | TiDB，TiKV，PD 监控聚合和上报端口 |
| Node_exporter | 9100 | TiDB 集群每个节点的系统信息上报通信端口 |
| Blackbox_exporter | 9115 | Blackbox_exporter 通信端口，用于 TiDB 集群端口监控 |
| Grafana | 3000 | Web 监控服务对外服务和客户端(浏览器)访问端口 |
| Grafana | 8686 | grafana_collector 通信端口，用于将 Dashboard 导出为 PDF 格式 |
| Kafka_exporter | 9308 | Kafka_exporter 通信端口，用于监控 binlog kafka 集群 |
三：环境部署：
3.1：群架构
xm-tidb-01 192.168.1.1
xm-pd-01 192.168.1.2
xm-tikv-01 192.168.1.3
xm-tikv-02 192.168.1.4
xm-tikv-03 192.168.1.5
3.2：创建ext4盘：（所有节点）
参照：https://www.cnblogs.com/jackyzm/p/10402275.html
vim mount-ext4.sh
#!/bin/sh
#https://www.cnblogs.com/jackyzm/p/10402275.html
##缩小home空间到5G
mkdir /homebak
sleep 1
cp -r /home /homebak
sleep 5
umount /home
lvremove /dev/mapper/centos-home -y
sleep 3
lvcreate -L 5G -n home centos -y
sleep 3
mkfs.xfs /dev/mapper/centos-home
sleep 15
mount /dev/mapper/centos-home /home
##新建ext4分区
lvcreate -L 20G -n ext4 centos -y
sleep 3
mkfs.ext4 /dev/mapper/centos-ext4
sleep 15
lsblk -f
./mount-ext4.sh
vim /etc/fstab 添加
UUID=a45530a9-8b07-4b9e-b78b-d6480e239dea /ext4 ext4 defaults,nodelalloc,noatime 0 2
mkdir /ext4
mount -a
mount -t ext4
3.3：安装依赖包（在主控机操作xm-tidb-01 192.168.1.1）
yum -y install epel-release git curl sshpass
yum install -y python-pip
升级pip
pip install --upgrade pip
[root@zz-01 /]# pip -V
pip 19.0.2 from /usr/lib/python2.7/site-packages/pip (python 2.7)
3.4：创建tidb用户
useradd -m -d /home/tidb tidb
passwd tidb
3.5：配置tidb用户sudo免密码
vim /etc/sudoers
添加 tidb ALL=(ALL) NOPASSWD:ALL 到末尾
用:wq!保持并推出
3.6：切换用户
su - tidb
3.6：创建tidb用户ssh key
ssh-keygen -t rsa
[tidb@xm-tidb-01 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tidb/.ssh/id_rsa):
Created directory '/home/tidb/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tidb/.ssh/id_rsa.
Your public key has been saved in /home/tidb/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:7oWdyfPJIGh10V8oJR3FB2BbSFJp9W3ZX4G/8leAl/4 tidb@xm-tidb-01
The key's randomart image is:
+---[RSA 2048]----+
| .o**B*.|
| ++*.oB|
| ..+.ooO|
| ..o++o|
| S . o..o|
| + = o ....|
| o + O o..|
| . . o = . .E|
| . + .|
+----[SHA256]-----+
四：在中控机部署TiDB-Ansible
4.1：各版本对应关系
| tidb-ansible 分支 | TiDB 版本 | 备注 |
|---|---|---|
| release-2.0 | 2.0 版本 | 最新 2.0 稳定版本，可用于生产环境。 |
| release-2.1 | 2.1 版本 | 最新 2.1 稳定版本，可用于生产环境（建议）。 |
| master | master 版本 | 包含最新特性，每日更新。 |
以tidb用户进入/home/tidb
4.2：下载对应tidb-ansible版本：（本例下载2.1版本）
release-2.0版本：git clone -b release-2.0 https://github.com/pingcap/tidb-ansible.git
release-2.1版本：git clone -b release-2.1 https://github.com/pingcap/tidb-ansible.git
master版本： git clone https://github.com/pingcap/tidb-ansible.git
4.3：安装ansible及其依赖：
cd /home/tidb/tidb-ansible/
确定pip版本为19.0.2以上pip -V
[tidb@xm-tidb-01 tidb-ansible]$ pip -V
pip 19.0.2 from /usr/lib/python2.7/site-packages/pip (python 2.7)
sudo pip install -r ./requirements.txt
ansible --version
[tidb@xm-tidb-01 tidb-ansible]$ ansible --version
ansible 2.6.13
config file = /home/tidb/tidb-ansible/ansible.cfg
configured module search path = [u'/home/tidb/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /bin/ansible
python version = 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
五：部署集群ssh互信及sudo规则：（tidb用户在中控机上操作）
5.1：添加列表
cd /home/tidb/tidb-ansible/
vim hosts.ini
[servers]
192.168.10.221
192.168.10.222
192.168.10.223
192.168.10.224
192.168.10.225 [all:vars]
username = tidb
ntp_server = pool.ntp.org
5.2： 执行以下命令，按提示输入部署目标机器 root 用户密码。该步骤将在部署目标机器上创建 tidb 用户，并配置 sudo 规则，配置中控机与部署目标机器之间的 ssh 互信。
ansible-playbook -i hosts.ini create_users.yml -u root -k
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook -i hosts.ini create_users.yml -u root -k
SSH password: PLAY [all] ************************************************************************************************* TASK [create user] *****************************************************************************************
changed: [192.168.10.225]
changed: [192.168.10.224]
changed: [192.168.10.223]
changed: [192.168.10.222]
ok: [192.168.10.221] TASK [set authorized key] **********************************************************************************
changed: [192.168.10.221]
changed: [192.168.10.225]
changed: [192.168.10.224]
changed: [192.168.10.222]
changed: [192.168.10.223] TASK [update sudoers file] *********************************************************************************
changed: [192.168.10.221]
changed: [192.168.10.223]
changed: [192.168.10.224]
changed: [192.168.10.222]
changed: [192.168.10.225] PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=3 changed=2 unreachable=0 failed=0
192.168.10.222 : ok=3 changed=3 unreachable=0 failed=0
192.168.10.223 : ok=3 changed=3 unreachable=0 failed=0
192.168.10.224 : ok=3 changed=3 unreachable=0 failed=0
192.168.10.225 : ok=3 changed=3 unreachable=0 failed=0 Congrats! All goes well. :-)
六：在目标机上安装NTP服务
6.1：cd /home/tidb/tidb-ansible
6.2：ansible-playbook -i hosts.ini deploy_ntp.yml -u tidb -b
如机器未装ntp服务，脚本会自动安装
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook -i hosts.ini deploy_ntp.yml -u tidb -b PLAY [all] ************************************************************************************************* TASK [get facts] *******************************************************************************************
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.222]
ok: [192.168.10.223] TASK [RedHat family Linux distribution - make sure ntp, ntpstat have been installed] ***********************
changed: [192.168.10.221] => (item=[u'ntp'])
changed: [192.168.10.223] => (item=[u'ntp'])
changed: [192.168.10.224] => (item=[u'ntp'])
changed: [192.168.10.225] => (item=[u'ntp'])
changed: [192.168.10.222] => (item=[u'ntp']) TASK [RedHat family Linux distribution - make sure ntpdate have been installed] ****************************
ok: [192.168.10.221] => (item=[u'ntpdate'])
ok: [192.168.10.222] => (item=[u'ntpdate'])
ok: [192.168.10.223] => (item=[u'ntpdate'])
ok: [192.168.10.224] => (item=[u'ntpdate'])
ok: [192.168.10.225] => (item=[u'ntpdate']) TASK [Debian family Linux distribution - make sure ntp, ntpstat have been installed] *********************** TASK [Debian family Linux distribution - make sure ntpdate have been installed] **************************** TASK [RedHat family Linux distribution - make sure ntpd service has been stopped] **************************
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224] TASK [Debian family Linux distribution - make sure ntp service has been stopped] *************************** TASK [Adjust Time | start to adjust time with pool.ntp.org] ************************************************
changed: [192.168.10.222]
changed: [192.168.10.224]
changed: [192.168.10.223]
changed: [192.168.10.221]
changed: [192.168.10.225] TASK [RedHat family Linux distribution - make sure ntpd service has been started] **************************
changed: [192.168.10.221]
changed: [192.168.10.222]
changed: [192.168.10.223]
changed: [192.168.10.224]
changed: [192.168.10.225] TASK [Debian family Linux distribution - Make sure ntp service has been started] *************************** PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=6 changed=3 unreachable=0 failed=0
192.168.10.222 : ok=6 changed=3 unreachable=0 failed=0
192.168.10.223 : ok=6 changed=3 unreachable=0 failed=0
192.168.10.224 : ok=6 changed=3 unreachable=0 failed=0
192.168.10.225 : ok=6 changed=3 unreachable=0 failed=0 Congrats! All goes well. :-)
七：在部署目标机配置cpufreq调节器模式：
7.1查看调节器模式
cpupower frequency-info --governors
[tidb@xm-tidb-01 tidb-ansible]$ cpupower frequency-info --governors
analyzing CPU 0:
available cpufreq governors: Not Available
返回 “Not Available”，表示当前系统不支持配置 CPUfreq，跳过该步骤即可。
八：分配机器资源，编辑inventory.ini文件：
8.1：vim /home/tidb/tidb-ansible/inventory.ini
## TiDB Cluster Part
[tidb_servers]
192.168.10.221 [tikv_servers]
192.168.10.223
192.168.10.224
192.168.10.225 [pd_servers]
192.168.10.222 [spark_master] [spark_slaves] [lightning_server] [importer_server] ## Monitoring Part
# prometheus and pushgateway servers
[monitoring_servers]
192.168.10.221 [grafana_servers]
192.168.10.221 # node_exporter and blackbox_exporter servers
[monitored_servers]
192.168.10.221
192.168.10.222
192.168.10.223
192.168.10.224
192.168.10.225 [alertmanager_servers] [kafka_exporter_servers] ## Binlog Part
[pump_servers] [drainer_servers]
8.2：inventory.ini变量调整：
部署目录调整：
## Global variables
[all:vars]
#deploy_dir = /home/tidb/deploy
deploy_dir = /ext4/deploy
如为某一服务单独设置部署目录，可在配置服务主机列表时配置主机变量，以 TiKV 节点为例，其他服务类推，请务必添加第一列别名，以免服务混布时混淆
TiKV1-1 ansible_host=172.16.10.4 deploy_dir=/data1/deploy
8.3：其他变量调整：True、False首字母要大写
| 变量 | 含义 |
|---|---|
| cluster_name | 集群名称，可调整 |
| tidb_version | TiDB 版本，TiDB-Ansible 各分支默认已配置 |
| process_supervision | 进程监管方式，默认为 systemd，可选 supervise |
| timezone | 新安装 TiDB 集群第一次启动 bootstrap（初始化）时，将 TiDB 全局默认时区设置为该值。TiDB 使用的时区后续可通过 time_zone 全局变量和 session 变量来修改，参考时区支持。 默认为 Asia/Shanghai，可选值参考timzone 列表。 |
| enable_firewalld | 开启防火墙，默认不开启，如需开启，请将部署建议-网络要求 中的端口加入白名单 |
| enable_ntpd | 检测部署目标机器 NTP 服务，默认为 True，请勿关闭 |
| set_hostname | 根据 IP 修改部署目标机器主机名，默认为 False |
| enable_binlog | 是否部署 pump 并开启 binlog，默认为 False，依赖 Kafka 集群，参见 zookeeper_addrs 变量 |
| zookeeper_addrs | binlog Kafka 集群的 zookeeper 地址 |
| enable_slow_query_log | TiDB 慢查询日志记录到单独文件({{ deploy_dir }}/log/tidb_slow_query.log)，默认为 False，记录到 tidb 日志 |
| deploy_without_tidb | KV 模式，不部署 TiDB 服务，仅部署 PD、TiKV 及监控服务，请将 inventory.ini 文件中 tidb_servers 主机组 IP 设置为空。 |
| alertmanager_target | 可选：如果你已单独部署 alertmanager，可配置该变量，格式：alertmanager_host:alertmanager_port |
| grafana_admin_user | Grafana 管理员帐号用户名，默认为 admin |
| grafana_admin_password | Grafana 管理员帐号密码，默认为 admin，用于 Ansible 导入 Dashboard 和创建 API Key，如后期通过 grafana web 修改了密码，请更新此变量 |
| collect_log_recent_hours | 采集日志时，采集最近几个小时的日志，默认为 2 小时 |
| enable_bandwidth_limit | 在中控机上从部署目标机器拉取诊断数据时，是否限速，默认为 True，与 collect_bandwidth_limit 变量结合使用 |
| collect_bandwidth_limit | 在中控机上从部署目标机器拉取诊断数据时限速多少，单位: Kbit/s，默认 10000，即 10Mb/s，如果是单机多 TiKV 实例部署方式，需除以单机实例个数 |
九：部署任务：
9.1：确认inventory.ini中ansible_user = tidb
## Connection
# ssh via normal user
ansible_user = tidb
9.2：测试ssh互信
ansible -i inventory.ini all -m shell -a 'whoami'
[tidb@xm-tidb-01 tidb-ansible]$ ansible -i inventory.ini all -m shell -a 'whoami'
192.168.10.224 | SUCCESS | rc=0 >>
tidb 192.168.10.223 | SUCCESS | rc=0 >>
tidb 192.168.10.221 | SUCCESS | rc=0 >>
tidb 192.168.10.222 | SUCCESS | rc=0 >>
tidb 192.168.10.225 | SUCCESS | rc=0 >>
tidb
9.3：测试sudo免密
ansible -i inventory.ini all -m shell -a 'whoami' -b
[tidb@xm-tidb-01 tidb-ansible]$ ansible -i inventory.ini all -m shell -a 'whoami' -b
192.168.10.224 | SUCCESS | rc=0 >>
root 192.168.10.223 | SUCCESS | rc=0 >>
root 192.168.10.221 | SUCCESS | rc=0 >>
root 192.168.10.222 | SUCCESS | rc=0 >>
root 192.168.10.225 | SUCCESS | rc=0 >>
root
9.4：下载TiDB binary到中控机：
ansible-playbook local_prepare.yml
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook local_prepare.yml PLAY [do local preparation] ******************************************************************************** TASK [local : Stop if ansible version is too low, make sure that the Ansible version is Ansible 2.4.2 or later, otherwise a compatibility issue occurs.] ***
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
} TASK [local : create downloads and resources directories] **************************************************
changed: [localhost] => (item=/home/tidb/tidb-ansible/downloads)
changed: [localhost] => (item=/home/tidb/tidb-ansible/resources)
changed: [localhost] => (item=/home/tidb/tidb-ansible/resources/bin) TASK [local : create cert directory] *********************************************************************** TASK [local : create packages.yml] *************************************************************************
changed: [localhost] TASK [local : create specific deployment method packages.yml] **********************************************
changed: [localhost] TASK [local : include_vars] ********************************************************************************
ok: [localhost] TASK [local : include_vars] ********************************************************************************
ok: [localhost] TASK [local : detect outbound network] *********************************************************************
ok: [localhost] TASK [local : set outbound network fact] *******************************************************************
ok: [localhost] TASK [local : fail] **************************************************************************************** TASK [local : detect GFW] **********************************************************************************
ok: [localhost] TASK [local : set GFW fact] ********************************************************************************
ok: [localhost] TASK [local : download tidb binary] ************************************************************************
FAILED - RETRYING: download tidb binary (4 retries left).
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-v2.1.4-linux-amd64.tar.gz', u'version': u'v2.1.4', u'name': u'tidb'}) TASK [local : download common binary] **********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/fio-3.8.tar.gz', u'checksum': u'sha256:15739abde7e74b59ac59df57f129b14fc5cd59e1e2eca2ce37b41f8c289c3d58', u'version': 3.8, u'name': u'fio'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/grafana_collector-latest-linux-amd64.tar.gz', u'version': u'latest', u'name': u'grafana_collector'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/kafka_exporter-1.1.0.linux-amd64.tar.gz', u'version': u'1.1.0', u'name': u'kafka_exporter'}) TASK [local : download diagnosis tools] ********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-insight-v0.2.5-1-g99b8fea.tar.gz', u'version': u'v0.2.5-1-g99b8fea', u'name': u'tidb-insight'}) TASK [local : download cfssl binary] *********************************************************************** TASK [local : download cfssljson binary] ******************************************************************* TASK [local : include_tasks] *******************************************************************************
included: /home/tidb/tidb-ansible/roles/local/tasks/binary_deployment.yml for localhost TASK [local : download other binary] *********************************************************************** TASK [local : download other binary under gfw] *************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/prometheus-2.2.1.linux-amd64.tar.gz', u'version': u'2.2.1', u'name': u'prometheus'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/alertmanager-0.14.0.linux-amd64.tar.gz', u'version': u'0.14.0', u'name': u'alertmanager'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/node_exporter-0.15.2.linux-amd64.tar.gz', u'version': u'0.15.2', u'name': u'node_exporter'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/pushgateway-0.4.0.linux-amd64.tar.gz', u'version': u'0.4.0', u'name': u'pushgateway'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/grafana-4.6.3.linux-x64.tar.gz', u'version': u'4.6.3', u'name': u'grafana'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/blackbox_exporter-0.12.0.linux-amd64.tar.gz', u'version': u'0.12.0', u'name': u'blackbox_exporter'}) TASK [local : download TiSpark packages] *******************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/spark-2.3.2-bin-hadoop2.7.tgz', u'checksum': u'sha256:6246b20d95c7596a29fb26d5b50a3ae3163a35915bec6c515a8e183383bedc43', u'version': u'2.3.2', u'name': u'spark-2.3.2-bin-hadoop2.7.tgz'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tispark-latest-linux-amd64.tar.gz', u'version': u'latest', u'name': u'tispark-latest.tar.gz'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tispark-sample-data.tar.gz', u'version': u'latest', u'name': u'tispark-sample-data.tar.gz'}) TASK [local : unarchive third party binary] ****************************************************************
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/prometheus/releases/download/v2.2.1/prometheus-2.2.1.linux-amd64.tar.gz', u'version': u'2.2.1', u'name': u'prometheus'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/alertmanager/releases/download/v0.14.0/alertmanager-0.14.0.linux-amd64.tar.gz', u'version': u'0.14.0', u'name': u'alertmanager'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz', u'version': u'0.15.2', u'name': u'node_exporter'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/blackbox_exporter/releases/download/v0.12.0/blackbox_exporter-0.12.0.linux-amd64.tar.gz', u'version': u'0.12.0', u'name': u'blackbox_exporter'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/pushgateway/releases/download/v0.4.0/pushgateway-0.4.0.linux-amd64.tar.gz', u'version': u'0.4.0', u'name': u'pushgateway'})
changed: [localhost] => (item={u'url': u'https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.3.linux-x64.tar.gz', u'version': u'4.6.3', u'name': u'grafana'}) TASK [local : unarchive tispark] ***************************************************************************
changed: [localhost] TASK [local : unarchive tispark-sample-data] ***************************************************************
changed: [localhost] TASK [local : cp monitoring binary] ************************************************************************
changed: [localhost] => (item=alertmanager)
changed: [localhost] => (item=prometheus)
changed: [localhost] => (item=node_exporter)
changed: [localhost] => (item=pushgateway)
changed: [localhost] => (item=blackbox_exporter) TASK [local : cp tispark] **********************************************************************************
changed: [localhost] TASK [local : cp tispark-sample-data] **********************************************************************
changed: [localhost] TASK [local : unarchive tidb binary] ***********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-v2.1.4-linux-amd64.tar.gz', u'version': u'v2.1.4', u'name': u'tidb'}) TASK [local : unarchive common binary] *********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/fio-3.8.tar.gz', u'checksum': u'sha256:15739abde7e74b59ac59df57f129b14fc5cd59e1e2eca2ce37b41f8c289c3d58', u'version': 3.8, u'name': u'fio'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/grafana_collector-latest-linux-amd64.tar.gz', u'version': u'latest', u'name': u'grafana_collector'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/kafka_exporter-1.1.0.linux-amd64.tar.gz', u'version': u'1.1.0', u'name': u'kafka_exporter'}) TASK [local : cp tidb binary] ******************************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-v2.1.4-linux-amd64.tar.gz', u'version': u'v2.1.4', u'name': u'tidb'}) TASK [local : cp fio binary] *******************************************************************************
changed: [localhost] => (item=fio) TASK [local : cp grafana_collector binary and fonts] *******************************************************
changed: [localhost] TASK [local : cp kafka_exporter binary] ********************************************************************
changed: [localhost] => (item=kafka_exporter) TASK [local : cp daemontools binary] *********************************************************************** TASK [local : cp tidb-insight tarball] *********************************************************************
changed: [localhost] TASK [local : clean up download dir] ***********************************************************************
changed: [localhost] PLAY RECAP *************************************************************************************************
localhost : ok=30 changed=22 unreachable=0 failed=0 Congrats! All goes well. :-)
[tidb@xm-tidb-01 tidb-ansible]$
9.5：初始化系统环境
ansible-playbook bootstrap.yml
报错：内存不足
TASK [check_system_optional : Preflight check - Check TiDB server's RAM] ***********************************
fatal: [192.168.10.221]: FAILED! => {"changed": false, "msg": "This machine does not have sufficient RAM to
run TiDB, at least 16000 MB."} NO MORE HOSTS LEFT *****************************************************************************************
to retry, use: --limit @/home/tidb/tidb-ansible/retry_files/bootstrap.retry PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=29 changed=10 unreachable=0 failed=1
192.168.10.222 : ok=29 changed=10 unreachable=0 failed=0
192.168.10.223 : ok=29 changed=10 unreachable=0 failed=0
192.168.10.224 : ok=29 changed=10 unreachable=0 failed=0
192.168.10.225 : ok=29 changed=10 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0 ERROR MESSAGE SUMMARY **************************************************************************************
[192.168.10.221]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight
check - Check TiDB server's RAM; message: {"changed": false, "msg": "This machine does not have sufficient
RAM to run TiDB, at least 16000 MB."} Ask for help:
Contact us: support@pingcap.com
It seems that you encounter some problems. You can send an email to the above email address, attached with
the tidb-ansible/inventory.ini and tidb-ansible/log/ansible.log files and the error message, or new issue
on https://github.com/pingcap/tidb-ansible/issues. We'll try our best to help you deploy a TiDB cluster.
Thanks. :-)
[tidb@xm-tidb-01 tidb-ansible]$
修改文件：
vim bootstrap.yml
注销掉：
- { role: check_system_optional, when: not dev_mode|default(false) }
- { role: machine_benchmark, when: not dev_mode|default(false) }
- name: check system
hosts: all
any_errors_fatal: true
roles:
- check_system_static
# - { role: check_system_optional, when: not dev_mode|default(false) } - name: tikv_servers machine benchmark
hosts: tikv_servers
gather_facts: false
roles:
# - { role: machine_benchmark, when: not dev_mode|default(false) }
再次运行：
ansible-playbook bootstrap.yml
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook bootstrap.yml PLAY [initializing deployment target] ********************************************************************** TASK [check_config_static : Ensure only one monitoring host exists] **************************************** TASK [check_config_static : Ensure monitored_servers exists] *********************************************** TASK [check_config_static : Ensure TiDB host exists] ******************************************************* TASK [check_config_static : Ensure PD host exists] ********************************************************* TASK [check_config_static : Ensure TiKV host exists] ******************************************************* TASK [check_config_static : Check ansible_user variable] *************************************************** TASK [check_config_static : Ensure timezone variable is set] *********************************************** TASK [check_config_static : Close old SSH control master processes] ****************************************
ok: [localhost] PLAY [check node config] *********************************************************************************** TASK [pre-ansible : disk space check - fail when disk is full] *********************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [pre-ansible : Get distro name from /etc/os-release] **************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [pre-ansible : set distro facts] **********************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [pre-ansible : python check] **************************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [pre-ansible : set has_python facts] ******************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [pre-ansible : set has_python facts] ****************************************************************** TASK [pre-ansible : include_tasks] ************************************************************************* TASK [pre-ansible : include_tasks] *************************************************************************
included: /home/tidb/tidb-ansible/roles/pre-ansible/tasks/root_tasks.yml for 192.168.10.221, 192.168.10.222, 192.168.10.223, 192.168.10.224, 192.168.10.225 TASK [pre-ansible : Debian/Ubuntu - install python] ******************************************************** TASK [pre-ansible : Redhat/CentOS - install python] ******************************************************** TASK [pre-ansible : Redhat/CentOS - Make sure ntp, ntpstat have been installed] ****************************
ok: [192.168.10.221] => (item=[u'ntp'])
ok: [192.168.10.222] => (item=[u'ntp'])
ok: [192.168.10.224] => (item=[u'ntp'])
ok: [192.168.10.223] => (item=[u'ntp'])
ok: [192.168.10.225] => (item=[u'ntp']) TASK [pre-ansible : Debian/Ubuntu - Make sure ntp, ntpstat have been installed] **************************** TASK [bootstrap : gather facts] ****************************************************************************
ok: [192.168.10.225]
ok: [192.168.10.222]
ok: [192.168.10.221]
ok: [192.168.10.223]
ok: [192.168.10.224] TASK [bootstrap : group hosts by distribution] *************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [bootstrap : Set deploy_dir if not presented] ********************************************************* TASK [bootstrap : include_tasks] ***************************************************************************
included: /home/tidb/tidb-ansible/roles/bootstrap/tasks/root_tasks.yml for 192.168.10.221, 192.168.10.222, 192.168.10.223, 192.168.10.224, 192.168.10.225 TASK [bootstrap : setting absent kernel params] ************************************************************
ok: [192.168.10.222] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.225] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.223] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.221] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.224] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0}) TASK [bootstrap : setting present kernel params] ***********************************************************
ok: [192.168.10.221] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.223] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.224] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.225] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.222] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.221] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.223] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.225] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.224] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.222] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.221] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.223] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.225] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.224] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.222] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.221] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.223] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.224] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.222] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.225] => (item={u'name': u'fs.file-max', u'value': 1000000}) TASK [bootstrap : update /etc/security/limits.conf] ********************************************************
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.223]
ok: [192.168.10.225]
ok: [192.168.10.222] TASK [bootstrap : disable swap] **************************************************************************** TASK [bootstrap : create group] ****************************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.224]
ok: [192.168.10.225]
ok: [192.168.10.223] TASK [bootstrap : create account] **************************************************************************
ok: [192.168.10.222]
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.225]
ok: [192.168.10.223] TASK [bootstrap : create top deploy dir when under root] ***************************************************
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.223]
ok: [192.168.10.222]
ok: [192.168.10.225] TASK [bootstrap : create wal_dir deploy dir when under root] *********************************************** TASK [bootstrap : create raftdb_path deploy dir when under root] ******************************************* TASK [bootstrap : set hostname if hostname is not distinguishable] ***************************************** TASK [bootstrap : set hostname in hosts file] ************************************************************** TASK [bootstrap : determine if firewalld is running] *******************************************************
ok: [192.168.10.223]
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.222] TASK [bootstrap : disable firewalld] *********************************************************************** TASK [bootstrap : or to enable firewalld] ****************************************************************** TASK [bootstrap : check centos configuration file exists] **************************************************
ok: [192.168.10.222]
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.223] TASK [bootstrap : check debian configuration file exists] **************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [bootstrap : modify centos irqbalance configuration file] *********************************************
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.223]
ok: [192.168.10.225]
ok: [192.168.10.222] TASK [bootstrap : modify debian irqbalance configuration file] ********************************************* TASK [bootstrap : start irqbalance service] ****************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.224]
ok: [192.168.10.225]
ok: [192.168.10.223] PLAY [check system] **************************************************************************************** TASK [check_system_static : Disk space check - Fail task when disk is full] ********************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225] TASK [check_system_static : get facts] *********************************************************************
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.222]
ok: [192.168.10.223] TASK [check_system_static : Preflight check - Linux OS family and distribution version] ******************** TASK [check_system_static : Deploy check_cpufreq script] ***************************************************
changed: [192.168.10.221]
changed: [192.168.10.224]
changed: [192.168.10.223]
changed: [192.168.10.225]
changed: [192.168.10.222] TASK [check_system_static : Preflight check - Check CPUfreq governors available in the kernel] *************
changed: [192.168.10.221]
changed: [192.168.10.222]
changed: [192.168.10.223]
changed: [192.168.10.224]
changed: [192.168.10.225] TASK [check_system_static : Preflight check - Check the currently active governor] *************************
changed: [192.168.10.224]
changed: [192.168.10.223]
changed: [192.168.10.222]
changed: [192.168.10.225]
changed: [192.168.10.221] TASK [check_system_static : Preflight check - Fail when CPU frequency governor is not set to performance mode] *** TASK [check_system_static : Clean check_cpufreq script] ****************************************************
changed: [192.168.10.221]
changed: [192.168.10.222]
changed: [192.168.10.223]
changed: [192.168.10.224]
changed: [192.168.10.225] TASK [check_system_static : Preflight check - Check Linux kernel overcommit_memory parameter] **************
changed: [192.168.10.221]
changed: [192.168.10.223]
changed: [192.168.10.222]
changed: [192.168.10.225]
changed: [192.168.10.224] TASK [check_system_static : Preflight check - Fail when Linux kernel vm.overcommit_memory parameter is set to 2] *** PLAY [tikv_servers machine benchmark] ********************************************************************** PLAY [create ops scripts] ********************************************************************************** TASK [ops : create check_tikv.sh script] *******************************************************************
changed: [localhost] TASK [ops : create pd-ctl.sh script] ***********************************************************************
changed: [localhost] PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.222 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.223 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.224 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.225 : ok=28 changed=5 unreachable=0 failed=0
localhost : ok=3 changed=2 unreachable=0 failed=0 Congrats! All goes well. :-)
9.6：部署TiDB集群软件：
ansible-playbook deploy.yml
过程比较多，请耐心等待
PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=115 changed=60 unreachable=0 failed=0
192.168.10.222 : ok=52 changed=23 unreachable=0 failed=0
192.168.10.223 : ok=60 changed=24 unreachable=0 failed=0
192.168.10.224 : ok=60 changed=24 unreachable=0 failed=0
192.168.10.225 : ok=63 changed=26 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0 Congrats! All goes well. :-)
[tidb@xm-tidb-01 tidb-ansible]$
注：Grafana Dashboard 上的 Report 按钮可用来生成 PDF 文件，此功能依赖 fontconfig 包和英文字体。如需使用该功能，登录 grafana_servers 机器，用以下命令安装：
sudo yum install fontconfig open-sans-fonts
9.7：启动集群：
！！！先切换到tidb用户！！！
su tidb
cd /home/tidb/tidb-ansible
ansible-playbook start.yml
PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=33 changed=12 unreachable=0 failed=0
192.168.10.222 : ok=12 changed=3 unreachable=0 failed=0
192.168.10.223 : ok=14 changed=3 unreachable=0 failed=0
192.168.10.224 : ok=14 changed=3 unreachable=0 failed=0
192.168.10.225 : ok=14 changed=3 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0 Congrats! All goes well. :-)
[tidb@xm-tidb-01 tidb-ansible]$
9.8： 测试集群：
在安装有mysql的其他服务器做连接测试
mysql -u root -h 192.168.10.221 -P 4000
-u和-h后的参数间不可有空格，否则会报错
mysql -uroot -pxxx -h 192.168.10.221 -P 4000
[root@zabbix ~]# mysql -u root -h 192.168.10.221 -P 4000
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 51
Server version: 5.7.10-TiDB-v2.1.4 MySQL Community Server (Apache License 2.0) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>
http://192.168.10.221:3000/
admin admin
感谢：
官方文档:
https://pingcap.com/docs-cn/overview/
https://blog.csdn.net/xujiamin0022016/article/details/83507038