Postgres-XL集群软件介绍及搭建

2024-01-06 21:48:58

介绍Postgres-XL

Postgres-XL 全称为 Postgres eXtensible Lattice，是TransLattice公司及其收购数据库技术公司–StormDB的产品。Postgres-XL是一个横向扩展的开源数据库集群，具有足够的灵活性来处理不同的数据库任务。

Postgres-XL功能特性

开放源代码：（源协议使用宽松的“Mozilla Public License”许可，允许将开源代码与闭源代码混在一起使用。）
完全的ACID支持
可横向扩展的关系型数据库（RDBMS）
- 支持OLAP应用，采用MPP（Massively Parallel Processing：大规模并行处理系统）架构模式
- 支持OLTP应用，读写性能可扩展
- 集群级别的ACID特性
- 多租户安全
- 也可被用作分布式Key-Value存储
事务处理与数据分析处理混合型数据库
支持丰富的SQL语句类型，比如：关联子查询
支持绝大部分PostgreSQL的SQL语句
分布式多版本并发控制（MVCC：Multi-version Concurrency Control）
支持JSON和XML格式

Postgres-XL缺少的功能

内建的高可用机制
- 使用外部机制实现高可能，如：Corosync/Pacemaker
- 有未来功能提升的空间
增加节点/重新分片数据（re-shard）的简便性
- 数据重分布（redistribution）期间会锁表
- 可采用预分片（pre-shard）方式解决，在同台物理服务器上建立多个数据节点，每个节点存储一个数据分片。数据重分布时，将一些数据节点迁出即可
某些外键、唯一性约束功能

Postgres-XL架构

基于开源项目Postgres-XC
- XL增加了MPP，允许数据节点间直接通讯，交换复杂跨节点关联查询相关数据信息，减少协调器负载。
多个协调器（Coordinator）
- 应用程序的数据库连入点
- 分析查询语句，生成执行计划
多个数据节点（DataNode）
- 实际的数据存储
- 数据自动打散分布到集群中各数据节点
- 本地执行查询
- 一个查询在所有相关节点上并行查询
全局事务管理器（GTM：Global Transaction Manager）
- 提供事务间一致性视图
- 部署GTM Proxy实例，以提高性能

Postgre-XL主要组件

GTM (Global Transaction Manager) - 全局事务管理器
GTM是Postgres-XL的一个关键组件，用于提供一致的事务管理和元组可见性控制。
GTM Standby
GTM的备节点，在pgxc,pgxl中，GTM控制所有的全局事务分配，如果出现问题，就会导致整个集群不可用，为了增加可用性，增加该备用节点。当GTM出现问题时，GTM Standby可以升级为GTM，保证集群正常工作。
GTM-Proxy
GTM需要与所有的Coordinators通信，为了降低压力，可以在每个Coordinator机器上部署一个GTM-Proxy。
Coordinator --协调器
协调器是应用程序到数据库的接口。它的作用类似于传统的PostgreSQL后台进程，但是协调器不存储任何实际数据。实际数据由数据节点存储。协调器接收SQL语句，根据需要获取全局事务Id和全局快照，确定涉及哪些数据节点，并要求它们执行(部分)语句。当向数据节点发出语句时，它与GXID和全局快照相关联，以便多版本并发控制(MVCC)属性扩展到集群范围。
Datanode --数据节点
用于实际存储数据。表可以分布在各个数据节点之间，也可以复制到所有数据节点。数据节点没有整个数据库的全局视图，它只负责本地存储的数据。接下来，协调器将检查传入语句，并制定子计划。然后，根据需要将这些数据连同GXID和全局快照一起传输到涉及的每个数据节点。数据节点可以在不同的会话中接收来自各个协调器的请求。但是，由于每个事务都是惟一标识的，并且与一致的(全局)快照相关联，所以每个数据节点都可以在其事务和快照上下文中正确执行。

Postgres-XL继承了PostgreSQL

Postgres-XL是PostgreSQL的扩展并继承了其很多特性：

复杂查询
外键
触发器
视图
事务
MVCC(多版本控制)

此外，类似于PostgreSQL，用户可以通过多种方式扩展Postgres-XL，例如添加新的

数据类型
函数
操作
聚合函数
索引类型
过程语言

安装

i## 环境说明
由于资源有限，gtm一台、另外两台身兼数职。

主机名	IP	角色	端口	nodename	数据目录
gtm	192.168.20.132	GTM	6666	gtm	/nodes/gtm
		协调器	5432	coord1	/nodes/coordinator
xl1	192.168.20.133	数据节点	5433	node1	/nodes/pgdata
		gtm代理	6666	gtmpoxy01	/nodes/gtm_pxy1
		协调器	5432	coord2	/nodes/coordinator
xl2	192.168.20.134	数据节点	5433	node2	/nodes/pgdata
		gtm代理	6666	gtmpoxy02	/nodes/gtm_pxy2

要求

GNU make版本 3.8及以上版本

[root@pg ~]# make --version
GNU Make 3.82
Built for x86_64-redhat-linux-gnu
Copyright (C) 2010  Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

需安装GCC包
需安装tar包
用于解压缩文件
默认需要GNU Readline library
其作用是可以让psql命令行记住执行过的命令，并且可以通过键盘上下键切换命令。但是可以通过--without-readline禁用这个特性，或者可以指定--withlibedit-preferred选项来使用libedit
默认使用zlib压缩库
可通过--without-zlib选项来禁用

配置hosts

所有主机上都配置

[root@xl2 11]# cat /etc/hosts
127.0.0.1   localhost
192.168.20.132  gtm
192.168.20.133  xl1
192.168.20.134  xl2

关闭防火墙、Selinux

所有主机都执行
关闭防火墙：

[root@gtm ~]# systemctl stop firewalld.service
[root@gtm ~]# systemctl disable firewalld.service

selinux设置:

[root@gtm ~]#vim /etc/selinux/config

设置SELINUX=disabled，保存退出。

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.

安装依赖包

所有主机上都执行

yum install -y flex bison readline-devel zlib-devel openjade docbook-style-dsssl  gcc

创建用户

所有主机上都执行

[root@gtm ~]# useradd postgres
[root@gtm ~]# passwd postgres
[root@gtm ~]# su - postgres
[root@gtm ~]# mkdir ~/.ssh
[root@gtm ~]# chmod 700 ~/.ssh

配置SSH免密登录
仅仅在gtm节点配置如下操作：

[root@gtm ~]# su - postgres
[postgres@gtm ~]# ssh-keygen -t rsa
[postgres@gtm ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[postgres@gtm ~]# chmod 600 ~/.ssh/authorized_keys

将刚生成的认证文件拷贝到xl1到xl2中，使得gtm节点可以免密码登录xl1~xl2的任意一个节点：

[postgres@gtm ~]# scp ~/.ssh/authorized_keys postgres@xl1:~/.ssh/
[postgres@gtm ~]# scp ~/.ssh/authorized_keys postgres@xl2:~/.ssh/

对所有提示都不要输入，直接enter下一步。直到最后，因为第一次要求输入目标机器的用户密码，输入即可。

下载源码

下载地址：https://www.postgres-xl.org/download/

[root@slave ~]# ll postgres-xl-10r1.1.tar.gz
-rw-r--r-- 1 root root 28121666 May 30 05:21 postgres-xl-10r1.1.tar.gz

编译、安装Postgres-XL

所有节点都安装，编译需要一点时间，最好同时进行编译。

[root@slave ~]# tar xvf postgres-xl-10r1.1.tar.gz
[root@slave ~]# ./configure --prefix=/home/postgres/pgxl/
[root@slave ~]# make
[root@slave ~]# make install
[root@slave ~]# cd contrib/   --安装必要的工具,在gtm节点上安装即可
[root@slave ~]# make
[root@slave ~]# make install

配置环境变量

所有节点都要配置
进入postgres用户，修改其环境变量，开始编辑

[root@gtm ~]#su - postgres
[postgres@gtm ~]#vi .bashrc    --不是.bash_profile

在打开的文件末尾，新增如下变量配置：

export PGHOME=/home/postgres/pgxl
export LD_LIBRARY_PATH=$PGHOME/lib:$LD_LIBRARY_PATH
export PATH=$PGHOME/bin:$PATH

按住esc，然后输入:wq!保存退出。输入以下命令对更改重启生效。

[postgres@gtm ~]# source .bashrc   --不是.bash_profile

输入以下语句，如果输出变量结果，代表生效

[postgres@gtm ~]# echo $PGHOME

应该输出/home/postgres/pgxl代表生效

配置集群

生成pgxc_ctl.conf配置文件

[postgres@gtm ~]# pgxc_ctl prepare
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxl/pgxc_ctl/pgxc_ctl_bash.
ERROR: File "/home/postgres/pgxl/pgxc_ctl/pgxc_ctl.conf" not found or not a regular file. No such file or directory
Installing pgxc_ctl_bash script as /home/postgres/pgxl/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxl/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxl/pgxc_ctl --configuration /home/postgres/pgxl/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxl/pgxc_ctl

配置pgxc_ctl.conf

新建/home/postgres/pgxc_ctl/pgxc_ctl.conf文件，编辑如下：

对着模板文件一个一个修改，否则会造成初始化过程出现各种神奇问题。

pgxcInstallDir=$PGHOME
pgxlDATA=$PGHOME/data 

pgxcOwner=postgres

#---- GTM Master -----------------------------------------
gtmName=gtm
gtmMasterServer=gtm
gtmMasterPort=6666
gtmMasterDir=$pgxlDATA/nodes/gtm

gtmSlave=y                  # Specify y if you configure GTM Slave.   Otherwise, GTM slave will not be configured and
                            # all the following variables will be reset.
gtmSlaveName=gtmSlave
gtmSlaveServer=gtm      # value none means GTM slave is not available.  Give none if you don't configure GTM Slave.
gtmSlavePort=20001          # Not used if you don't configure GTM slave.
gtmSlaveDir=$pgxlDATA/nodes/gtmSlave    # Not used if you don't configure GTM slave.

#---- GTM-Proxy Master -------
gtmProxyDir=$pgxlDATA/nodes/gtm_proxy
gtmProxy=y                              
gtmProxyNames=(gtm_pxy1 gtm_pxy2)   
gtmProxyServers=(xl1 xl2)           
gtmProxyPorts=(6666 6666)               
gtmProxyDirs=($gtmProxyDir $gtmProxyDir)            

#---- Coordinators ---------
coordMasterDir=$pgxlDATA/nodes/coord
coordNames=(coord1 coord2)      
coordPorts=(5432 5432)          
poolerPorts=(6667 6667)         
coordPgHbaEntries=(0.0.0.0/0)

coordMasterServers=(xl1 xl2)    
coordMasterDirs=($coordMasterDir $coordMasterDir)
coordMaxWALsernder=0    #没设置备份节点，设置为0
coordMaxWALSenders=($coordMaxWALsernder $coordMaxWALsernder) #数量保持和coordMasterServers一致

coordSlave=n

#---- Datanodes ----------
datanodeMasterDir=$pgxlDATA/nodes/dn_master
primaryDatanode=xl1               # 主数据节点
datanodeNames=(node1 node2)
datanodePorts=(5433 5433)   
datanodePoolerPorts=(6668 6668) 
datanodePgHbaEntries=(0.0.0.0/0)

datanodeMasterServers=(xl1 xl2)
datanodeMasterDirs=($datanodeMasterDir $datanodeMasterDir)
datanodeMaxWalSender=4
datanodeMaxWALSenders=($datanodeMaxWalSender $datanodeMaxWalSender)

集群初始化，启动，停止

初始化

pgxc_ctl -c /home/postgres/pgxc_ctl/pgxc_ctl.conf init all

输出结果：

/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
/home/postgres/pgxc_ctl/pgxc_ctl.conf: line 189: $coordExtraConfig: ambiguous redirect
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
Stopping all the coordinator masters.
Stopping coordinator master coord1.
Stopping coordinator master coord2.
pg_ctl: directory "/home/postgres/pgxc/nodes/coord/coord1" does not exist
pg_ctl: directory "/home/postgres/pgxc/nodes/coord/coord2" does not exist
Done.
Stopping all the datanode masters.
Stopping datanode master datanode1.
Stopping datanode master datanode2.
pg_ctl: PID file "/home/postgres/pgxc/nodes/datanode/datanode1/postmaster.pid" does not exist
Is server running?
Done.
Stop GTM master
waiting for server to shut down.... done
server stopped
[postgres@gtm ~]$ echo $PGHOME
/home/postgres/pgxl
[postgres@gtm ~]$ ll /home/postgres/pgxl/pgxc/nodes/gtm/gtm.^C
[postgres@gtm ~]$ pgxc_ctl -c /home/postgres/pgxc_ctl/pgxc_ctl.conf init all
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
/home/postgres/pgxc_ctl/pgxc_ctl.conf: line 189: $coordExtraConfig: ambiguous redirect
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
Initialize GTM master
ERROR: target directory (/home/postgres/pgxc/nodes/gtm) exists and not empty. Skip GTM initilialization
Done.
Start GTM master
server starting
Initialize all the coordinator masters.
Initialize coordinator master coord1.
ERROR: target coordinator master coord1 is running now.   Skip initilialization.
Initialize coordinator master coord2.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/coord/coord2 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
Done.
Starting coordinator master.
Starting coordinator master coord1
ERROR: target coordinator master coord1 is already running now.   Skip initialization.
Starting coordinator master coord2
2019-05-30 21:09:25.562 EDT [2148] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-05-30 21:09:25.562 EDT [2148] LOG:  listening on IPv6 address "::", port 5432
2019-05-30 21:09:25.563 EDT [2148] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-05-30 21:09:25.601 EDT [2149] LOG:  database system was shut down at 2019-05-30 21:09:22 EDT
2019-05-30 21:09:25.605 EDT [2148] LOG:  database system is ready to accept connections
2019-05-30 21:09:25.612 EDT [2156] LOG:  cluster monitor started
Done.
Initialize all the datanode masters.
Initialize the datanode master datanode1.
Initialize the datanode master datanode2.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/datanode/datanode1 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/datanode/datanode2 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
Done.
Starting all the datanode masters.
Starting datanode master datanode1.
WARNING: datanode master datanode1 is running now. Skipping.
Starting datanode master datanode2.
2019-05-30 21:09:33.352 EDT [2404] LOG:  listening on IPv4 address "0.0.0.0", port 15432
2019-05-30 21:09:33.352 EDT [2404] LOG:  listening on IPv6 address "::", port 15432
2019-05-30 21:09:33.355 EDT [2404] LOG:  listening on Unix socket "/tmp/.s.PGSQL.15432"
2019-05-30 21:09:33.392 EDT [2404] LOG:  redirecting log output to logging collector process
2019-05-30 21:09:33.392 EDT [2404] HINT:  Future log output will appear in directory "pg_log".
Done.
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
Done.
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
Done.
[postgres@gtm ~]$ pgxc_ctl -c /home/postgres/pgxc_ctl/pgxc_ctl.conf stop all
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
/home/postgres/pgxc_ctl/pgxc_ctl.conf: line 189: $coordExtraConfig: ambiguous redirect
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
Stopping all the coordinator masters.
Stopping coordinator master coord1.
Stopping coordinator master coord2.
pg_ctl: directory "/home/postgres/pgxc/nodes/coord/coord1" does not exist
Done.
Stopping all the datanode masters.
Stopping datanode master datanode1.
Stopping datanode master datanode2.
pg_ctl: PID file "/home/postgres/pgxc/nodes/datanode/datanode1/postmaster.pid" does not exist
Is server running?
Done.
Stop GTM master
waiting for server to shut down.... done
server stopped
[postgres@gtm ~]$ pgxc_ctl
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
/home/postgres/pgxc_ctl/pgxc_ctl.conf: line 189: $coordExtraConfig: ambiguous redirect
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
PGXC monitor all
Not running: gtm master
Running: coordinator master coord1
Not running: coordinator master coord2
Running: datanode master datanode1
Not running: datanode master datanode2
PGXC stop coordinator master coord1
Stopping coordinator master coord1.
pg_ctl: directory "/home/postgres/pgxc/nodes/coord/coord1" does not exist
Done.
PGXC stop datanode master datanode1
Stopping datanode master datanode1.
pg_ctl: PID file "/home/postgres/pgxc/nodes/datanode/datanode1/postmaster.pid" does not exist
Is server running?
Done.
PGXC monitor all
Not running: gtm master
Running: coordinator master coord1
Not running: coordinator master coord2
Running: datanode master datanode1
Not running: datanode master datanode2
PGXC monitor all
Not running: gtm master
Not running: coordinator master coord1
Not running: coordinator master coord2
Not running: datanode master datanode1
Not running: datanode master datanode2
PGXC exit
[postgres@gtm ~]$ pgxc_ctl -c /home/postgres/pgxc_ctl/pgxc_ctl.conf init all
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
/home/postgres/pgxc_ctl/pgxc_ctl.conf: line 189: $coordExtraConfig: ambiguous redirect
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
Initialize GTM master
ERROR: target directory (/home/postgres/pgxc/nodes/gtm) exists and not empty. Skip GTM initilialization
Done.
Start GTM master
server starting
Initialize all the coordinator masters.
Initialize coordinator master coord1.
Initialize coordinator master coord2.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/coord/coord1 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/coord/coord2 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
Done.
Starting coordinator master.
Starting coordinator master coord1
Starting coordinator master coord2
2019-05-30 21:13:03.998 EDT [25137] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-05-30 21:13:03.998 EDT [25137] LOG:  listening on IPv6 address "::", port 5432
2019-05-30 21:13:04.000 EDT [25137] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-05-30 21:13:04.038 EDT [25138] LOG:  database system was shut down at 2019-05-30 21:13:00 EDT
2019-05-30 21:13:04.042 EDT [25137] LOG:  database system is ready to accept connections
2019-05-30 21:13:04.049 EDT [25145] LOG:  cluster monitor started
2019-05-30 21:13:04.020 EDT [2730] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-05-30 21:13:04.020 EDT [2730] LOG:  listening on IPv6 address "::", port 5432
2019-05-30 21:13:04.021 EDT [2730] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-05-30 21:13:04.057 EDT [2731] LOG:  database system was shut down at 2019-05-30 21:13:00 EDT
2019-05-30 21:13:04.061 EDT [2730] LOG:  database system is ready to accept connections
2019-05-30 21:13:04.062 EDT [2738] LOG:  cluster monitor started
Done.
Initialize all the datanode masters.
Initialize the datanode master datanode1.
Initialize the datanode master datanode2.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/datanode/datanode1 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgxc/nodes/datanode/datanode2 ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... creating cluster information ... ok
syncing data to disk ... ok
freezing database template0 ... ok
freezing database template1 ... ok
freezing database postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success.
Done.
Starting all the datanode masters.
Starting datanode master datanode1.
Starting datanode master datanode2.
2019-05-30 21:13:12.077 EDT [25392] LOG:  listening on IPv4 address "0.0.0.0", port 15432
2019-05-30 21:13:12.077 EDT [25392] LOG:  listening on IPv6 address "::", port 15432
2019-05-30 21:13:12.079 EDT [25392] LOG:  listening on Unix socket "/tmp/.s.PGSQL.15432"
2019-05-30 21:13:12.114 EDT [25392] LOG:  redirecting log output to logging collector process
2019-05-30 21:13:12.114 EDT [25392] HINT:  Future log output will appear in directory "pg_log".
2019-05-30 21:13:12.079 EDT [2985] LOG:  listening on IPv4 address "0.0.0.0", port 15432
2019-05-30 21:13:12.079 EDT [2985] LOG:  listening on IPv6 address "::", port 15432
2019-05-30 21:13:12.081 EDT [2985] LOG:  listening on Unix socket "/tmp/.s.PGSQL.15432"
2019-05-30 21:13:12.117 EDT [2985] LOG:  redirecting log output to logging collector process
2019-05-30 21:13:12.117 EDT [2985] HINT:  Future log output will appear in directory "pg_log".
Done.
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
Done.
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
psql: FATAL:  no pg_hba.conf entry for host "192.168.20.132", user "postgres", database "postgres"
Done.

启动

pgxc_ctl -c /home/postgres/pgxc_ctl/pgxc_ctl.conf start all

关闭

pgxc_ctl -c /home/postgres/pgxc_ctl/pgxc_ctl.conf stop all

查看集群状态

[postgres@gtm ~]$ pgxc_ctl monitor all
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
/home/postgres/pgxc_ctl/pgxc_ctl.conf: line 189: $coordExtraConfig: ambiguous redirect
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
Running: gtm master
Running: coordinator master coord1
Running: coordinator master coord2
Running: datanode master datanode1
Running: datanode master datanode2

配置集群信息

分别在数据节点、协调器节点上分别执行以下命令：

注：本节点只执行修改操作即可(alert node)，其他节点执行创建命令(create node)。因为本节点已经包含本节点的信息。

create node coord1 with (type=coordinator,host=xl1, port=5432);
create node coord2 with (type=coordinator,host=xl2, port=5432);
alter node coord1 with (type=coordinator,host=xl1, port=5432);
alter node coord2 with (type=coordinator,host=xl2, port=5432);

create node datanode1 with (type=datanode, host=xl1,port=15432,primary=true,PREFERRED);
create node  datanode2 with (type=datanode, host=xl2,port=15432);
alter node  datanode1 with (type=datanode, host=xl1,port=15432,primary=true,PREFERRED);
alter node  datanode2 with (type=datanode, host=xl2,port=15432);
select pgxc_pool_reload();

分别登陆数据节点、协调器节点验证

postgres=# select * from pgxc_node;
 node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |   node_id
-----------+-----------+-----------+-----------+----------------+------------------+-------------
 coord1    | C         |      5432 | xl1       | f              | f                |  1885696643
 coord2    | C         |      5432 | xl2       | f              | f                | -1197102633
 datanode2 | D         |     15432 | xl2       | f              | f                |  -905831925
 datanode1 | D         |     15432 | xl1       | t              | f                |   888802358
(4 rows)

测试

插入数据

在数据节点1，执行相关操作。
通过协调器端口登录PG

[postgres@xl1 ~]$ psql -p 5432
psql (PGXL 10r1.1, based on PG 10.6 (Postgres-XL 10r1.1))
Type "help" for help.

postgres=# create database lei;
CREATE DATABASE

postgres=# \c lei;
You are now connected to database "lei" as user "postgres".
lei=# create table test1(id int,name text);
CREATE TABLE
lei=# insert into test1(id,name) select generate_series(1,8),'测试';
INSERT 0 8

lei=# select * from test1;
 id | name
----+------
  1 | 测试
  2 | 测试
  5 | 测试
  6 | 测试
  8 | 测试
  3 | 测试
  4 | 测试
  7 | 测试
(8 rows)

注：默认创建的表为分布式表，也就是每个数据节点值存储表的部分数据。关于表类型具体说明，下面有说明。

通过15432端口登录数据节点，查看数据
有5条数据

[postgres@xl1 ~]$ psql -p 15432
psql (PGXL 10r1.1, based on PG 10.6 (Postgres-XL 10r1.1))
Type "help" for help.

postgres=# \c lei;
You are now connected to database "lei" as user "postgres".
lei=# select * from test1;
 id | name
----+------
  1 | 测试
  2 | 测试
  5 | 测试
  6 | 测试
  8 | 测试
(5 rows)

登录到节点2，查看数据
有3条数据

[postgres@xl2 ~]$ psql -p15432
psql (PGXL 10r1.1, based on PG 10.6 (Postgres-XL 10r1.1))
Type "help" for help.

postgres=# \c lei;
You are now connected to database "lei" as user "postgres".
lei=# select * from test1;
 id | name
----+------
  3 | 测试
  4 | 测试
  7 | 测试
(3 rows)

两个节点的数据加起来整个8条，没有问题。

至此Postgre-XL集群搭建完成。

创建数据库、表时可能会出现以下错误：

ERROR: Failed to get pooled connections
是因为pg_hba.conf配置不对，所有节点加上host all all 192.168.20.0/0 trust并重启集群即可。
ERROR: No Datanode defined in cluster
首先确认是否创建了数据节点，也就是create node相关的命令。如果创建了则执行select pgxc_pool_reload();使其生效即可。

集群管理与应用

表类型说明

REPLICATION表：各个datanode节点中，表的数据完全相同，也就是说，插入数据时，会分别在每个datanode节点插入相同数据。读数据时，只需要读任意一个datanode节点上的数据。
建表语法：
```
CREATE TABLE repltab (col1 int, col2 int) DISTRIBUTE BY REPLICATION;
```
DISTRIBUTE ：会将插入的数据，按照拆分规则，分配到不同的datanode节点中存储，也就是sharding技术。每个datanode节点只保存了部分数据，通过coordinate节点可以查询完整的数据视图。
```
CREATE TABLE disttab(col1 int, col2 int, col3 text) DISTRIBUTE BY HASH(col1);
```

模拟数据插入
#任意登录一个coordinate节点进行建表操作

[postgres@gtm ~]$  psql -h  xl1 -p 5432 -U postgres
postgres=# INSERT INTO disttab SELECT generate_series(1,100), generate_series(101, 200), 'foo';
INSERT 0 100
postgres=# INSERT INTO repltab SELECT generate_series(1,100), generate_series(101, 200);
INSERT 0 100

查看数据分布结果：

DISTRIBUTE表分布结果

postgres=# SELECT xc_node_id, count(*) FROM disttab GROUP BY xc_node_id;
 xc_node_id | count 
------------+-------
 1148549230 |    42
 -927910690 |    58
(2 rows)

REPLICATION表分布结果

postgres=# SELECT count(*) FROM repltab;
 count 
-------
   100
(1 row)

查看另一个datanode2中repltab表结果

[postgres@datanode2 pgxl9.5]$ psql -p 15432
psql (PGXL 10r1.1, based on PG 10.6 (Postgres-XL 10r1.1))
Type "help" for help.

postgres=# SELECT count(*) FROM repltab;
 count 
-------
   100
(1 row)

结论：REPLICATION表中，datanode1,datanode2中表是全部数据，一模一样。而DISTRIBUTE表，数据散落近乎平均分配到了datanode1,datanode2节点中。

新增数据节点与数据重分布

在线新增节点、并重新分布数据。

新增datanode节点

在gtm集群管理节点上执行pgxc_ctl命令

[postgres@gtm ~]$ pgxc_ctl
/bin/bash
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Installing pgxc_ctl_bash script as /home/postgres/pgxc_ctl/pgxc_ctl_bash.
Reading configuration using /home/postgres/pgxc_ctl/pgxc_ctl_bash --home /home/postgres/pgxc_ctl --configuration /home/postgres/pgxc_ctl/pgxc_ctl.conf
Finished reading configuration.
   ******** PGXC_CTL START ***************

Current directory: /home/postgres/pgxc_ctl
PGXC 

# 在服务器xl3上，新增一个master角色的datanode节点，名称是datanode3
# 端口号暂定5430，pool master暂定6669 ，指定好数据目录位置，从两个节点升级到3个节点，之后要写3个none
# none应该是datanodeSpecificExtraConfig或者datanodeSpecificExtraPgHba配置

PGXC add datanode master datanode3 xl3 15432 6671 /home/postgres/pgxc/nodes/datanode/datanode3 none none none

等待新增完成后，查询集群节点状态：

postgres=# select * from pgxc_node;
 node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |   node_id
-----------+-----------+-----------+-----------+----------------+------------------+-------------
 datanode1 | D         |     15432 | xl1       | t              | f                |   888802358
 datanode2 | D         |     15432 | xl2       | f              | f                |  -905831925
 datanode3 | D         |     15432 | xl3       | f              | f                |  -705831925
 coord1    | C         |      5432 | xl1       | f              | f                |  1885696643
 coord2    | C         |      5432 | xl2       | f              | f                | -1197102633
(4 rows)

节点新增完毕

数据重新分布

由于新增节点后无法自动完成数据重新分布，需要手动操作。
DISTRIBUTE表分布在了node1,node2节点上，如下：

postgres=# SELECT xc_node_id, count(*) FROM disttab GROUP BY xc_node_id;
 xc_node_id | count 
------------+-------
 1148549230 |    42
 -927910690 |    58
(2 rows)

新增一个节点后，将sharding表数据重新分配到三个节点上，将repl表复制到新节点

# 重分布sharding表
postgres=# ALTER TABLE disttab ADD NODE (datanode3);
ALTER TABLE
# 复制数据到新节点
postgres=#  ALTER TABLE repltab ADD NODE (datanode3);
ALTER TABLE

查看新的数据分布：

postgres=# SELECT xc_node_id, count(*) FROM disttab GROUP BY xc_node_id;
 xc_node_id | count 
------------+-------
 -700122826 |    36
 -927910690 |    32
 1148549230 |    32
(3 rows)

登录datanode3(新增的时候，放在了xl3服务器上，端口15432)节点查看数据：

[postgres@gtm ~]$ psql -h xl3 -p 15432 -U postgres
psql (PGXL 10r1.1, based on PG 10.6 (Postgres-XL 10r1.1))
Type "help" for help.
postgres=# select count(*) from repltab;
 count 
-------
   100
(1 row)

很明显,通过 ALTER TABLE tt ADD NODE (dn)命令，可以将DISTRIBUTE表数据重新分布到新节点，重分布过程中会中断所有事务。可以将REPLICATION表数据复制到新节点。

从datanode节点中回收数据

postgres=# ALTER TABLE disttab DELETE NODE (datanode3);
ALTER TABLE
postgres=# ALTER TABLE repltab DELETE NODE (datanode3);
ALTER TABLE

删除数据节点

Postgresql-XL并没有检查将被删除的datanode节点是否有replicated/distributed表的数据，为了数据安全，在删除之前需要检查下被删除节点上的数据，有数据的话，要回收掉分配到其他节点，然后才能安全删除。删除数据节点分为四步骤：

1.查询要删除节点dn3的oid

postgres=#  SELECT oid, * FROM pgxc_node;
oid  | node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |   node_id   
-------+-----------+-----------+-----------+-----------+----------------+------------------+-------------
11819 | coord1    | C         |      5432 | datanode1 | f              | f                |  1885696643
16384 | coord2    | C         |      5432 | datanode2 | f              | f                | -1197102633
16385 | node1     | D         |      5433 | datanode1 | f              | t                |  1148549230
16386 | node2     | D         |      5433 | datanode2 | f              | f                |  -927910690
16397 | dn3       | D         |      5430 | datanode1 | f              | f                |  -700122826
(5 rows)

2.查询dn3对应的oid中是否有数据

testdb=# SELECT * FROM pgxc_class WHERE nodeoids::integer[] @> ARRAY[16397];
pcrelid | pclocatortype | pcattnum | pchashalgorithm | pchashbuckets |     nodeoids      
---------+---------------+----------+-----------------+---------------+-------------------
   16388 | H             |        1 |               1 |          4096 | 16397 16385 16386
   16394 | R             |        0 |               0 |             0 | 16397 16385 16386
(2 rows)

3.有数据的先回收数据

postgres=# ALTER TABLE disttab DELETE NODE (dn3);
ALTER TABLE
postgres=# ALTER TABLE repltab DELETE NODE (dn3);
ALTER TABLE
postgres=# SELECT * FROM pgxc_class WHERE nodeoids::integer[] @> ARRAY[16397];
pcrelid | pclocatortype | pcattnum | pchashalgorithm | pchashbuckets | nodeoids 
---------+---------------+----------+-----------------+---------------+----------
(0 rows)

4.安全删除dn3

PGXC$  remove datanode master dn3 clean

故障节点FAILOVER

1.查看当前集群状态

 [postgres@gtm ~]$ psql -h  xl1 -p 5432
 psql (PGXL 10r1.1, based on PG 10.6 (Postgres-XL 10r1.1))
 Type "help" for help.

 postgres=# SELECT oid, * FROM pgxc_node;
 oid  | node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |   node_id
 -------+-----------+-----------+-----------+-----------+----------------+------------------+-------------
 11739 | coord1    | C         |      5432 | xl1       | f              | f                |  1885696643
 16384 | coord2    | C         |      5432 | xl2       | f              | f                | -1197102633
 16387 | datanode2 | D         |     15432 | xl2       | f              | f                |  -905831925
 16388 | datanode1 | D         |     15432 | xl1       | t              | t                |   888802358
 (4 rows)

2.模拟datanode1节点故障
直接关闭即可

 PGXC stop -m immediate datanode master datanode1
 Stopping datanode master datanode1.
 Done.

3.测试查询
只要查询涉及到datanode1上的数据，那么该查询就会报错

 postgres=# SELECT xc_node_id, count(*) FROM disttab GROUP BY xc_node_id;
 WARNING:  failed to receive file descriptors for connections
 ERROR:  Failed to get pooled connections
 HINT:  This may happen because one or more nodes are currently unreachable, either because of node or network failure.
 Its also possible that the target node may have hit the connection limit or the pooler is configured with low connections.
 Please check if all nodes are running fine and also review max_connections and max_pool_size configuration parameters
 postgres=# SELECT xc_node_id, * FROM disttab WHERE col1 = 3;
 xc_node_id | col1 | col2 | col3
 ------------+------+------+------
 -905831925 |    3 |  103 | foo
 (1 row)

测试发现，查询范围如果涉及到故障的node1节点，会报错，而查询的数据范围不在node1上的话，仍然可以查询。

4.手动切换
要想切换，必须要提前配置slave节点。

 PGXC$  failover datanode node1

切换完成后，查询集群

 postgres=# SELECT oid, * FROM pgxc_node;
 oid  | node_name | node_type | node_port | node_host | nodeis_primary | nodeis_preferred |   node_id   
 -------+-----------+-----------+-----------+-----------+----------------+------------------+-------------
 11819 | coord1    | C         |      5432 | datanode1 | f              | f                |  1885696643
 16384 | coord2    | C         |      5432 | datanode2 | f              | f                | -1197102633
 16386 | node2     | D         |      15432 | datanode2 | f              | f                |  -927910690
 16385 | node1     | D         |     15433 | datanode2 | f              | t                |  1148549230
 (4 rows)

发现datanode1节点的ip和端口都已经替换为配置的slave了。

码农公寓