使用Ansible部署etcd 3.2高可用集群

2022-09-02 22:50:08

之前写过一篇手动搭建etcd 3.1集群的文章《etcd 3.1 高可用集群搭建》，最近要初始化一套新的环境，考虑用ansible自动化部署整套环境，先从部署etcd 3.2集群开始。

需要部署etcd的主机信息如下：

node1 192.168.61.11

node2 192.168.61.12

node3 192.168.61.13

配置管理项目目录结构

├── inventories

│   ├── staging

│   │   ├── group_vars

│   │   │   ├── all.yml

│   │   │   └── etcd-nodes.yml

│   │   ├── host_vars

│   │   │   ├── node1.yml

│   │   │   ├── node2.yml

│   │   │   └── node3.yml

│   │   └── hosts

│   └── production

├── roles

│   ├── common

│   │   ├── defaults

│   │   │   └── main.yml

│   │   └── tasks

│   │       └── main.yml

│   ├── etcd3

│       ├── defaults

│       │   └── main.yml

│       ├── files

│       │   └── make-ca-cert.sh

│       ├── meta

│       │   └── main.yml

│       ├── tasks

│       │   ├── create_etcd_user.yml

│       │   ├── etcd-restart.yml

│       │   ├── etcd-start.yml

│       │   ├── etcd-stop.yml

│       │   ├── gen-etcd-certs.yml

│       │   ├── gen-etcd-systemd.yml

│       │   ├── install_etcd_bin.yml

│       │   └── main.yml

│       └── templates

│           ├── etcd.conf.j2

│           └── etcd.service.j2

├── deploy-etcd3.yml

roles/etcd3/defaults/main.yml：

---

etcd_version: 3.2.0

etcd_download_url_base: "https://github.com/coreos/etcd/releases/download/v{{ etcd_version }}"

etcd_release: "etcd-v{{ etcd_version }}-linux-amd64"

etcd_download_url: "{{ etcd_download_url_base }}/{{ etcd_release}}.tar.gz"

etcd_bin_path: /usr/bin

etcd_data_dir: /var/lib/etcd

etcd_conf_dir: /etc/etcd

etcd_certs_dir: "{{ etcd_conf_dir }}/ssl"

etcd_cert_group: root

etcd_ca_file: "{{ etcd_certs_dir }}/ca.crt"

etcd_cert_file: "{{ etcd_certs_dir }}/server.crt"

etcd_key_file: "{{ etcd_certs_dir }}/server.key"

etcd_peer_ca_file: "{{ etcd_certs_dir }}/ca.crt"

etcd_peer_cert_file: "{{ etcd_certs_dir }}/peer.crt"

etcd_peer_key_file: "{{ etcd_certs_dir }}/peer.key"

etcd_client_cert_file: "{{ etcd_certs_dir }}/client.crt"

etcd_client_key_file: "{{ etcd_certs_dir }}/client.key"

etcd_client_cert_auth: true

etcd_peer_client_cert_auth: true

etcd_client_port: 2379

etcd_peer_port: 2380

etcd_initial_cluster_state: new

etcd_initial_cluster_token: etcd-k8s-cluster

etcd_initial_advertise_peer_urls: "https://{{ etcd_machine_address }}:{{ etcd_peer_port }}"

etcd_listen_peer_urls: "https://{{ etcd_machine_address }}:{{ etcd_peer_port }}"

etcd_advertise_client_urls: "https://{{ etcd_machine_address }}:{{ etcd_client_port }}"

etcd_listen_client_urls: "https://{{ etcd_machine_address }}:2379,https://127.0.0.1:2379"

创建etcd用户和数据目录

创建etcd用户、用户组和数据目录。

- name: create system etcd group

  group:

    name: etcd

    state: present

- name: create system etcd user

  user:

    name: etcd

    comment: "etcd user"

    shell: /sbin/nologin

    state: present

    system: yes

    home: "{{ etcd_data_dir }}"

    groups: etcd

- name: ensure etcd_data_dir exists

  file:

    path: "{{ etcd_data_dir }}"

    recurse: yes

    state: directory

    owner: etcd

    group: etcd

下载和解压etcd

下载和解压缩etcd release tar包，并将可执行文件etcd, etcdctl拷贝到/usr/bin。

---

- name: set github s3 host on the first etcd server

  lineinfile:

    dest: /etc/hosts

    regexp: '.*github-production-release-asset-2e65be\.s3\.amazonaws\.com$'

    line: "219.76.4.4 github-production-release-asset-2e65be.s3.amazonaws.com"

    state: present

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  run_once: true

- name: check whether etcd release tar extracted on the first etcd server

  stat:

    path: "{{ ansible_temp_dir }}/{{ etcd_release }}"

  register: etcd_release_tar_check

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  run_once: true

- name: download etcd release tar file on first the etcd server

  get_url:

    url: "{{ etcd_download_url }}"

    dest: "{{ ansible_temp_dir }}"

    validate_certs: no

    timeout: 20

  register: download_etcd

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  run_once: true

  when: not etcd_release_tar_check.stat.exists

- name: extract etcd tar file

  unarchive:

    src: "{{ download_etcd.dest }}"

    dest: "{{ ansible_temp_dir }}"

    remote_src: yes

  run_once: true

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  when: not etcd_release_tar_check.stat.exists

- name: fetch etcd bins from the first etcd server

  fetch:

    src: "{{ ansible_temp_dir }}/{{ etcd_release }}/{{ item }}"

    dest: "tmp/etcd3/{{ item }}"

    flat: yes

  register: fetch_etcd

  run_once: true

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  with_items:

    - etcd

    - etcdctl

- name: copy etcd binary

  copy:

    src: "tmp/etcd3/{{ item }}"

    dest: "{{ etcd_bin_path }}"

    owner: etcd

    group: etcd

    mode: 0750

  with_items:

    - etcd

    - etcdctl

生成并分发etcd TLS证书

---

- name: ensure etcd certs directory

  file:

    path: "{{ etcd_certs_dir }}"

    state: directory

    owner: etcd

    group: etcd

    mode: 0750

    recurse: yes

- name: copy make-ca-cert.sh

  copy:

    src: make-ca-cert.sh

    dest: "{{ etcd_certs_dir }}"

    owner: root

    group: root

    mode: "0500"

  run_once: true

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

- name: gen certs on the first etcd server

  command:

    "{{ etcd_certs_dir }}/make-ca-cert.sh"

  args:

    creates: "{{ etcd_certs_dir }}/server.crt"

  run_once: true

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  environment:

    NODE_IPS: "{% for host in groups['etcd-nodes'] %}{{ hostvars[host]['etcd_machine_address'] }}{% if not loop.last %},{% endif %}{% endfor %}"

    NODE_DNS: "{{ groups['etcd-nodes']|join(',') }}"

    CERT_DIR: "{{ etcd_certs_dir }}"

    CERT_GROUP: "{{ etcd_cert_group }}"

- name: slurp etcd certs

  slurp:

    src: "{{ item }}"

  register: pki_certs

  run_once: true

  delegate_to: "{{ groups['etcd-nodes'][0] }}"

  with_items:

    - "{{ etcd_ca_file }}"

    - "{{ etcd_cert_file }}"

    - "{{ etcd_key_file }}"

    - "{{ etcd_peer_ca_file }}"

    - "{{ etcd_peer_cert_file }}"

    - "{{ etcd_peer_key_file }}"

    - "{{ etcd_client_cert_file }}"

    - "{{ etcd_client_key_file }}"

- name: copy etcd certs to other etcd servers

  copy:

    dest: "{{ item.item }}"

    content: "{{ item.content | b64decode }}"

    owner: etcd

    group: "{{ etcd_cert_group }}"

    mode: 0400

  with_items: "{{ pki_certs.results }}"

  when: inventory_hostname != groups['etcd-nodes'][0]

systemd和配置

---

- name: create etcd systemd unit file

  template:

    src: etcd.service.j2

    dest: /etc/systemd/system/etcd.service

- name: create etcd env conf

  template:

    src: etcd.conf.j2

    dest: /etc/etcd/etcd.conf

    owner: etcd

    group: etcd

    mode: 0540

启动etcd

---

- name: start etcd

  systemd:

    name: etcd

    daemon_reload: yes

    state: started

    enabled: yes

- name: restart etcd

  systemd:

    name: etcd

    state: restarted

查看集群状态

检查集群是否健康，在任一节点执行：

etcdctl \

  --ca-file=/etc/etcd/ssl/ca.crt \

  --cert-file=/etc/etcd/ssl/client.crt \

  --key-file=/etc/etcd/ssl/client.key \

  --endpoints=https://node1:2379,https://node2:2379,https://node3:2379 \

  cluster-health

member 1e3da2bf674fd07 is healthy: got healthy result from https://192.168.61.11:2379

member 88548a72a2e9a749 is healthy: got healthy result from https://192.168.61.13:2379

member c3bda13bf78ed2ab is healthy: got healthy result from https://192.168.61.12:2379

cluster is healthy

etcdctl \

  --ca-file=/etc/etcd/ssl/ca.crt \

  --cert-file=/etc/etcd/ssl/client.crt \

  --key-file=/etc/etcd/ssl/client.key \

  --endpoints=https://node1:2379,https://node2:2379,https://node3:2379 \

  member list

1e3da2bf674fd07: name=node1 peerURLs=https://192.168.61.11:2380 clientURLs=https://192.168.61.11:2379 isLeader=false

88548a72a2e9a749: name=node3 peerURLs=https://192.168.61.13:2380 clientURLs=https://192.168.61.13:2379 isLeader=false

c3bda13bf78ed2ab: name=node2 peerURLs=https://192.168.61.12:2380 clientURLs=https://192.168.61.12:2379 isLeader=true

标题：使用Ansible部署etcd 3.2高可用集群
本文链接：http://blog.frognew.com/2017/06/using-ansible-deploy-etcd-cluster.html
转载请注明出处。

← 团队环境：Jenkins版本升级和插件更新 → 使用Ansible部署Kubernetes 1.6高可用集群

码农公寓