背景
今天晚上同事给我请教了一个问题，gpreenplum数据库的gpadmin没有管理员权限了，我说是不是管理员权限被删除了，执行了 postgres --single alter user gpadmin superuser,后来执行gpstart -a报数据库无法启动,再后来就找我了
过程
无法创建用户,发现gpadmin用户没有superuser的权限
关闭数据库时发现只能关闭主节点
无法启动数据库
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[INFO]:-Starting gpstart with args: -a
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[INFO]:-Gathering information and validating the environment...
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.11.3 build 1'
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[WARNING]:-postmaster.pid file exists on Master, checking if recovery startup required
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[INFO]:-Commencing recovery startup checks
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[INFO]:-Have lock file /tmp/.s.PGSQL.5432 and a process running on port 5432
20210901:20:39:37:022692 gpstart:pmd:gpadmin-[ERROR]:-gpstart error: Master instance process running
20210901:20:41:44:023319 gpstart:pmd:gpadmin-[INFO]:-Starting gpstart with args: -a
20210901:20:41:44:023319 gpstart:pmd:gpadmin-[INFO]:-Gathering information and validating the environment...
20210901:20:41:44:023319 gpstart:pmd:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.11.3 build 1'
20210901:20:41:45:023319 gpstart:pmd:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'
20210901:20:41:45:023319 gpstart:pmd:gpadmin-[INFO]:-Starting Master instance in admin mode
20210901:20:41:46:023319 gpstart:pmd:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210901:20:41:46:023319 gpstart:pmd:gpadmin-[INFO]:-Obtaining Segment details from master...
20210901:20:41:46:023319 gpstart:pmd:gpadmin-[INFO]:-Setting new master era
20210901:20:41:46:023319 gpstart:pmd:gpadmin-[INFO]:-Master Started...
20210901:20:41:46:023319 gpstart:pmd:gpadmin-[INFO]:-Master start noticed recovered segments, updating configuration to rebalance.
20210901:20:41:46:023319 gpstart:pmd:gpadmin-[CRITICAL]:-gpstart failed. (Reason='error 'ERROR: permission denied for relation gp_segment_configuration
' in 'UPDATE gp_segment_configuration SET role=preferred_role where content = 0'') exiting... 这个原因是因为gpadmin已经没有超级管理员权限,导致无法在gp_segment_configuration更新mode和status,进而无法启动数据库
gp_segment_configuration需要被gpadmin(superuser权限)才可以修改，如果节点的数据库状态不对，需要更新gp_segment_configuration的status,role,mode的值因为没有权限，故而数据库无法启动
尝试利用复制文件替换
自己造了一些相似环境,备份要替换的文件后进行替换
20210901:21:33:35:004625 gpstart:pmd:gpadmin-[INFO]:-Starting gpstart with args: -a
20210901:21:33:35:004625 gpstart:pmd:gpadmin-[INFO]:-Gathering information and validating the environment...
20210901:21:33:35:004625 gpstart:pmd:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.11.3 build 1'
20210901:21:33:35:004625 gpstart:pmd:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'
20210901:21:33:35:004625 gpstart:pmd:gpadmin-[INFO]:-Starting Master instance in admin mode
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Obtaining Segment details from master...
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Setting new master era
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Master Started...
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Master start noticed recovered segments, updating configuration to rebalance.
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Re-obtaining updated segment details from master...
20210901:21:33:36:004625 gpstart:pmd:gpadmin-[INFO]:-Shutting down master
20210901:21:33:37:004625 gpstart:pmd:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-Process results...
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-----------------------------------------------------
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:- Successful segment starts = 8
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:- Failed segment starts = 0
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-----------------------------------------------------
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-Successfully started 8 of 8 segment instances
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-----------------------------------------------------
20210901:21:33:57:004625 gpstart:pmd:gpadmin-[INFO]:-Starting Master instance pmd directory /opt/master-data/master/gpseg-1
20210901:21:33:58:004625 gpstart:pmd:gpadmin-[INFO]:-Command pg_ctl reports Master pmd instance active
20210901:21:34:00:004625 gpstart:pmd:gpadmin-[WARNING]:-FATAL: DTM initialization: failure during startup recovery, retry failed, check segment status (cdbtm.c:1603)
20210901:21:34:00:004625 gpstart:pmd:gpadmin-[INFO]:-No standby master configured. skipping...
20210901:21:34:00:004625 gpstart:pmd:gpadmin-[INFO]:-Check status of database with gpstate utility 具体内容报错日志
2021-09-01 21:33:58.329217 CST,,,p4693,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: initializing XLog startup",,,,,,,0,,"primary_mirror_mode.c",1883,
2021-09-01 21:33:58.329376 CST,,,p4693,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: starting database",,,,,,,0,,"primary_mirror_mode.c",1897,
2021-09-01 21:33:58.331181 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","database system was shut down at 2021-09-01 21:33:36 CST",,,,,,,0,,"xlog.c",6371,
2021-09-01 21:33:58.331223 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint record is at 33/BBB45BE8",,,,,,,0,,"xlog.c",6559,
2021-09-01 21:33:58.331232 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","redo record is at 33/BBB45BE8; undo record is at 0/0; shutdown TRUE",,,,,,,0,,"xlog.c",6616,
2021-09-01 21:33:58.331241 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","next transaction ID: 0/26101623; next OID: 91210403",,,,,,,0,,"xlog.c",6620,
2021-09-01 21:33:58.331249 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","next MultiXactId: 144516; next MultiXactOffset: 724843",,,,,,,0,,"xlog.c",6623,
2021-09-01 21:33:58.331257 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 33/BBB45C90",,,,,,,0,,"xlog.c",6986,
tstore.c",350,
store.c",350,
ntstore.c",350,
tstore.c",350,
2021-09-01 21:33:58.349850 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","Oldest active transaction from prepared transactions 26101623",,,,,,,0,,"xlog.c",6033,
2021-09-01 21:33:58.374784 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",6084,
C) 4.4.2 compiled on Jan 24 2017 20:25:07",,,,,,,0,,"xlog.c",6095,
2021-09-01 21:33:58.381962 CST,,,p4695,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","Finished normal startup for clean shutdown case",,,,,,,0,,"xlog.c",7216,
2021-09-01 21:33:58.410000 CST,,,p4696,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","Finished BASIC startup integrity checking",,,,,,,0,,"xlog.c",7661,
C) 4.4.2 compiled on Jan 24 2017 20:27:17",,,,,,,0,,"postmaster.c",4618,
x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jan 24 2017 20:27:17",,,,,,0,,"postmaster.c",4624,
2021-09-01 21:33:58.415652 CST,,,p4694,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","SNMP inform/trap alerts are disabled",,,,,,,,"send_snmp_inform_or_trap","sendalert.c",228,
2021-09-01 21:33:58.415685 CST,,,p4694,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","e-mail alerts are disabled",,,,,,,,"send_alert_via_email","sendalert.c",569,
s:",,,,,,,0,,"cdbtm.c",3609,
s:",,,,,,,0,,"cdbtm.c",3609,
:40000",,,,,,,0,,"cdbgang.c",430,
g2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
because its gang was disconnected (session id = 5). The new session id = 6",,,,,,,0,,"cdbgang.c",2685,
:40000",,,,,,,0,,"cdbgang.c",430,
g2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
because its gang was disconnected (session id = 6). The new session id = 7",,,,,,,0,,"cdbgang.c",2685,
:40000",,,,,,,0,,"cdbgang.c",430,
g2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
because its gang was disconnected (session id = 7). The new session id = 8",,,,,,,0,,"cdbgang.c",2685,
:40000",,,,,,,0,,"cdbgang.c",430,
g2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
because its gang was disconnected (session id = 8). The new session id = 9",,,,,,,0,,"cdbgang.c",2685,
:40000",,,,,,,0,,"cdbgang.c",430,
g2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
t because its gang was disconnected (session id = 9). The new session id = 10",,,,,,,0,,"cdbgang.c",2685,
located",,,,,,,0,,"cdbgang.c",1662,
ception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
,,,,0,,"cdbtm.c",1582,
ns:",,,,,,,0,,"cdbtm.c",3609,
ns:",,,,,,,0,,"cdbtm.c",3609,
2:40000",,,,,,,0,,"cdbgang.c",430,
eg2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
t because its gang was disconnected (session id = 10). The new session id = 11",,,,,,,0,,"cdbgang.c",2685,
2:40000",,,,,,,0,,"cdbgang.c",430,
eg2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
t because its gang was disconnected (session id = 11). The new session id = 12",,,,,,,0,,"cdbgang.c",2685,
2:40000",,,,,,,0,,"cdbgang.c",430,
eg2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
t because its gang was disconnected (session id = 12). The new session id = 13",,,,,,,0,,"cdbgang.c",2685,
2:40000",,,,,,,0,,"cdbgang.c",430,
eg2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
t because its gang was disconnected (session id = 13). The new session id = 14",,,,,,,0,,"cdbgang.c",2685,
2:40000",,,,,,,0,,"cdbgang.c",430,
eg2 psd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
t because its gang was disconnected (session id = 14). The new session id = 15",,,,,,,0,,"cdbgang.c",2685,
located",,,,,,,0,,"cdbgang.c",1662,
ception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
,,,,0,,"cdbtm.c",1582,
retry.",,,,,,,0,,"cdbtm.c",1602,
2021-09-01 21:34:00.658837 CST,,,p4694,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /gp/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-22.base.el7.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
during startup recovery, retry failed, check segment status (cdbtm.c:1603)",,,,,,,0,,"cdbtm.c",1603,"Stack trace:
1 0xb06762 postgres <symbol not found> (elog.c:502)
2 0xb08778 postgres elog_finish (elog.c:1446)
3 0xcdaec2 postgres initTM (cdbtm.c:1497)
4 0xb1ec87 postgres InitPostgres (postinit.c:947)
5 0x99b04f postgres PostgresMain (postgres.c:4394)
6 0x8f955e postgres <symbol not found> (postmaster.c:6673)
7 0x8fc200 postgres PostmasterMain (postmaster.c:7604)
8 0x7fda6f postgres main (main.c:206)
9 0x7fd233aae3d5 libc.so.6 __libc_start_main + 0xf5
10 0x4bffe9 postgres <symbol not found> + 0x4bffe9
"
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
...
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
,,,,,,0,,"cdbtm.c",3609,
,,,,,,0,,"cdbtm.c",3609,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 28). The new session id = 29",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 29). The new session id = 30",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 30). The new session id = 31",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 31). The new session id = 32",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 32). The new session id = 33",,,,,,,0,,"cdbgang.c",2685,
ed",,,,,,,0,,"cdbgang.c",1662,
on: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
,,"cdbtm.c",1582,
,,,,,,0,,"cdbtm.c",3609,
,,,,,,0,,"cdbtm.c",3609,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 33). The new session id = 34",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 34). The new session id = 35",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 35). The new session id = 36",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 36). The new session id = 37",,,,,,,0,,"cdbgang.c",2685,
00",,,,,,,0,,"cdbgang.c",430,
sd2:40000 with options FATAL: could not open relation 1664/0/2676: No such file or directory",,,,,,,0,,"cdbgang.c",437,
ause its gang was disconnected (session id = 37). The new session id = 38",,,,,,,0,,"cdbgang.c",2685,
ed",,,,,,,0,,"cdbgang.c",1662,
on: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
,,"cdbtm.c",1582,
y.",,,,,,,0,,"cdbtm.c",1602,
2021-09-01 21:34:47.197196 CST,,,p4694,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /gp/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-22.base.el7.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
ng startup recovery, retry failed, check segment status (cdbtm.c:1603)",,,,,,,0,,"cdbtm.c",1603,"Stack trace:
1 0xb06762 postgres <symbol not found> (elog.c:502)
2 0xb08778 postgres elog_finish (elog.c:1446)
3 0xcdaec2 postgres initTM (cdbtm.c:1497)
4 0xb1ec87 postgres InitPostgres (postinit.c:947)
5 0x99b04f postgres PostgresMain (postgres.c:4394)
6 0x8f955e postgres <symbol not found> (postmaster.c:6673)
7 0x8fc200 postgres PostmasterMain (postmaster.c:7604)
8 0x7fda6f postgres main (main.c:206)
9 0x7fd233aae3d5 libc.so.6 __libc_start_main + 0xf5
10 0x4bffe9 postgres <symbol not found> + 0x4bffe9
"
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
...
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ation failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ation failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ntication failed for user ""only_select""",,,,,,,0,,"auth.c",326,
ion failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
ation failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
...
cation failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
cation failed for user ""gpadmin""",,,,,,,0,,"auth.c",326,
"BEGIN",0,,"postgres.c",1619,
IN_MESSAGES='ERROR'",,,,,,"SET CLIENT_MIN_MESSAGES='ERROR'",0,,"postgres.c",1619,
,"COMMIT",0,,"postgres.c",1619,
"BEGIN",0,,"postgres.c",1619,
on()",,,,,,"SELECT version()",0,,"postgres.c",1619,
me, typlen FROM pg_type WHERE oid=25",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=25",0,,"postgres.c",1619,
2021-09-01 22:21:44.609489 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd9,seg-1,,,x26102306,sx1,"LOG","00000","statement:
SELECT fault_strategy FROM gp_fault_strategy",,,,,,"
SELECT fault_strategy FROM gp_fault_strategy
",0,,"postgres.c",1619,
ame, typlen FROM pg_type WHERE oid=18",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=18",0,,"postgres.c",1619,
2021-09-01 22:21:44.611206 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd13,seg-1,,,x26102306,sx1,"LOG","00000","statement:
SELECT dbid, content, role, preferred_role, mode, status,
hostname, address, port, replication_port, fs.oid,
fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)
ORDER BY content, preferred_role DESC, fs.oid",,,,,,"
SELECT dbid, content, role, preferred_role, mode, status,
hostname, address, port, replication_port, fs.oid,
fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)
ORDER BY content, preferred_role DESC, fs.oid
",0,,"postgres.c",1619,
ame, typlen FROM pg_type WHERE oid=21",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=21",0,,"postgres.c",1619,
ame, typlen FROM pg_type WHERE oid=23",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=23",0,,"postgres.c",1619,
ame, typlen FROM pg_type WHERE oid=26",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=26",0,,"postgres.c",1619,
2021-09-01 22:21:44.617132 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd21,seg-1,,,x26102306,sx1,"LOG","00000","statement:
SELECT dbid, content, status, unnest(san_mounts)
FROM pg_catalog.gp_segment_configuration
WHERE content >= 0
ORDER BY content, dbid",,,,,,"
SELECT dbid, content, status, unnest(san_mounts)
FROM pg_catalog.gp_segment_configuration
WHERE content >= 0
ORDER BY content, dbid
",0,,"postgres.c",1619,
2021-09-01 22:21:44.617737 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd23,seg-1,,,x26102306,sx1,"LOG","00000","statement:
SELECT mountid, active_host, san_type,
primary_host, primary_mountpoint, primary_device,
mirror_host, mirror_mountpoint, mirror_device
FROM pg_catalog.gp_san_configuration
ORDER BY mountid",,,,,,"
SELECT mountid, active_host, san_type,
primary_host, primary_mountpoint, primary_device,
mirror_host, mirror_mountpoint, mirror_device
FROM pg_catalog.gp_san_configuration
ORDER BY mountid
",0,,"postgres.c",1619,
2021-09-01 22:21:44.618185 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd25,seg-1,,,x26102306,sx1,"LOG","00000","statement:
SELECT oid, fsname
FROM pg_filespace
ORDER BY fsname;",,,,,,"
SELECT oid, fsname
FROM pg_filespace
ORDER BY fsname;
",0,,"postgres.c",1619,
ame, typlen FROM pg_type WHERE oid=19",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=19",0,,"postgres.c",1619,
2021-09-01 22:21:44.620160 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd29,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 1",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 1
",0,,"postgres.c",1619,
2021-09-01 22:21:44.627165 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd31,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 2",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 2
",0,,"postgres.c",1619,
2021-09-01 22:21:44.634160 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd33,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 6",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 6
",0,,"postgres.c",1619,
2021-09-01 22:21:44.646825 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd35,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 3",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 3
",0,,"postgres.c",1619,
2021-09-01 22:21:44.648142 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd37,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 7",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 7
",0,,"postgres.c",1619,
2021-09-01 22:21:44.654175 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd39,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 4",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 4
",0,,"postgres.c",1619,
2021-09-01 22:21:44.662850 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd41,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 8",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 8
",0,,"postgres.c",1619,
2021-09-01 22:21:44.666147 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd43,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 5",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 5
",0,,"postgres.c",1619,
2021-09-01 22:21:44.672289 CST,"gpadmin","template1",p19035,th941422464,"::1","46660",2021-09-01 22:21:44 CST,26102306,con705,cmd45,seg-1,,,x26102306,sx1,"LOG","00000","statement:
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 9",,,,,,"
select fsloc.fselocation || '/' ||
case when db.dattablespace = 1663
then 'base'
else db.dattablespace::text
end || '/'||db.oid as catloc
from pg_Database db, pg_tablespace ts,
(SELECT dbid, fs.oid, fselocation
FROM pg_catalog.gp_segment_configuration
JOIN pg_catalog.pg_filespace_entry on (dbid = fsedbid)
JOIN pg_catalog.pg_filespace fs on (fsefsoid = fs.oid)) fsloc
where db.dattablespace = ts.oid
and ts.spcfsoid = fsloc.oid
and fsloc.dbid = 9
",0,,"postgres.c",1619,
"BEGIN",0,,"postgres.c",1619,
IN_MESSAGES='ERROR'",,,,,,"SET CLIENT_MIN_MESSAGES='ERROR'",0,,"postgres.c",1619,
,"COMMIT",0,,"postgres.c",1619,
"BEGIN",0,,"postgres.c",1619,
id FROM pg_stat_activity WHERE procpid != pg_backend_pid()",,,,,,"SELECT procpid FROM pg_stat_activity WHERE procpid != pg_backend_pid()",0,,"postgres.c",1619,
me, typlen FROM pg_type WHERE oid=23",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=23",0,,"postgres.c",1619,
2021-09-01 22:21:44.749130 CST,,,p4693,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","received fast shutdown request",,,,,,,0,,"postmaster.c",4384,
2021-09-01 22:21:44.754023 CST,,,p4698,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","shutting down",,,,,,,0,,"xlog.c",8502,
2021-09-01 22:21:44.765644 CST,,,p4698,th941422464,,,,0,,,seg-1,,,,,"LOG","00000","database system is shut down",,,,,,,0,,"xlog.c",8518,
当时看到这个时候首先因为是可能xmin,xmax大于现在的最大的xmax，导致报错，后来仔细观看日志报错，怀疑是gpadmin的密码有问题，因为密码是自己默认的gpadmin，而对方的密码我的环境不一致
不过当时考虑到还有一种方式来解决问题，就把之前替换的文件重新还原回去。
尝试利用命令替换
这里要注意一定要正常关闭gpstop -af，否则可能出现各种更加突出的情况。
每个数据节点(master,slave1,slave2)的segment的都要修改
$ /gp/greenplum-db-4.3.11.3/bin/postgres --single -P -O -D /opt/data/primary2/gpseg1/ -p 40001 -c gp_session_role=utility postgres
PostgreSQL stand-alone backend 8.2.15
backend> alter user gpadmin superuser 本次出现一个问题是slave1节点执行都出现问题
$ /gp/greenplum-db-4.3.11.3/bin/postgres --single -P -O -D /opt/data/primary2/gpseg1/ -p 40001 -c gp_session_role=utility postgres
2021-09-01 23:27:56.405543 CST,,,p28380,th-1909614720,,,,0,,,seg-1,,,,,"FATAL","XX000","Database must be shutdown cleanly when using single backend start (xlog.c:6688)",,,,,,,,"StartupXLOG","
xlog.c",6688,1 0xb0a95e postgres errstart (elog.c:502)2 0x565064 postgres StartupXLOG (xlog.c:6687)
3 0x99f16b postgres PostgresMain (postgres.c:4342)
4 0x7fdaff postgres main (main.c:198)
5 0x7f8389bba505 libc.so.6 __libc_start_main + 0xf5
6 0x4bffe9 postgres <symbol not found> + 0x4 后面就启动了(master,slave2)的master,primary,mirror后，启动成功了数据库(slave2的primary,mirror都启动了)
这里觉得做到对的情况，就是做了一 个启动后将数据备份出来了。
$ gpstart -a
...
2021-09-01 23:59:49.851268 CST,,,p16119,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: initializing XLog startup",,,,,,,0,,"primary_mirror_mode.c",1883,
2021-09-01 23:59:49.851424 CST,,,p16119,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: starting database",,,,,,,0,,"primary_mirror_mode.c",1897,
2021-09-01 23:59:49.853153 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","database system was shut down at 2021-09-01 23:59:46 CST",,,,,,,0,,"xlog.c",6371,
2021-09-01 23:59:49.853181 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint record is at 33/BBB52F68",,,,,,,0,,"xlog.c",6559,
2021-09-01 23:59:49.853191 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","redo record is at 33/BBB52F68; undo record is at 0/0; shutdown TRUE",,,,,,,0,,"xlog.c",6616,
2021-09-01 23:59:49.853204 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","next transaction ID: 0/26103277; next OID: 91210403",,,,,,,0,,"xlog.c",6620,
2021-09-01 23:59:49.853213 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","next MultiXactId: 144516; next MultiXactOffset: 724843",,,,,,,0,,"xlog.c",6623,
2021-09-01 23:59:49.853222 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 33/BBB53010",,,,,,,0,,"xlog.c",6986,
entstore.c",350,
ntstore.c",350,
tentstore.c",350,
entstore.c",350,
2021-09-01 23:59:49.878170 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","Oldest active transaction from prepared transactions 26103277",,,,,,,0,,"xlog.c",6033,
2021-09-01 23:59:49.898899 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",6084,
GCC) 4.4.2 compiled on Jan 24 2017 20:25:07",,,,,,,0,,"xlog.c",6095,
2021-09-01 23:59:49.905597 CST,,,p16121,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","Finished normal startup for clean shutdown case",,,,,,,0,,"xlog.c",7216,
2021-09-01 23:59:49.938224 CST,,,p16122,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","Finished BASIC startup integrity checking",,,,,,,0,,"xlog.c",7661,
GCC) 4.4.2 compiled on Jan 24 2017 20:27:17",,,,,,,0,,"postmaster.c",4618,
n x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jan 24 2017 20:27:17",,,,,,0,,"postmaster.c",4624,
2021-09-01 23:59:49.941814 CST,,,p16120,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","SNMP inform/trap alerts are disabled",,,,,,,,"send_snmp_inform_or_trap","sendalert.c",228,
2021-09-01 23:59:49.941847 CST,,,p16120,th1088092032,,,,0,,,seg-1,,,,,"LOG","00000","e-mail alerts are disabled",,,,,,,,"send_alert_via_email","sendalert.c",569,
ons:",,,,,,,0,,"cdbtm.c",3609,
ons:",,,,,,,0,,"cdbtm.c",3609,
ry for host ""173.16.91.47"", user ""gpadmin"", database ""bus_yyjk"", SSL off",,,,,,,0,,"auth.c",608,
id is 2",,,,,,,0,,"cdbtm.c",3122,
.c",3125,
eduler.c",266,
,,"BEGIN",0,,"postgres.c",1619,
_MIN_MESSAGES='ERROR'",,,,,,"SET CLIENT_MIN_MESSAGES='ERROR'",0,,"postgres.c",1619,
,,,"COMMIT",0,,"postgres.c",1619,
.....
备份数据库
$pg_dump dbname | gzip > abc.gz备份数据库的过程前误删节点/home/gpadmin目录
当时不小心将节点二的/home/gpadmin目录删除
将节点1的/home/gpadmin拷贝到节点二的/home/gpadmin下，主要是拷贝.ssh目录下信息
授权目录权
# chown -R gpadmin:gpadmin /home/gpadmin恢复数据库产生新问题
$ gprecoverseg 数据库修复后，数据库启动
$ gpstop -af$ gpstart -a 数据库启动失败
$
2021-09-02 23:04:14.673366 CST,,,p16553,th1111873408,,,,0,,,seg-1,,,,,"LOG","00000","received fast shutdown request",,,,,,,0,,"postmaster.c",4384,
2021-09-02 23:04:14.677110 CST,,,p16558,th1111873408,,,,0,,,seg-1,,,,,"LOG","00000","shutting down",,,,,,,0,,"xlog.c",8502,
2021-09-02 23:04:14.686227 CST,,,p16558,th1111873408,,,,0,,,seg-1,,,,,"LOG","00000","database system is shut down",,,,,,,0,,"xlog.c",8518,
2021-09-02 23:04:24.202522 CST,,,p16607,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: initializing XLog startup",,,,,,,0,,"primary_mirror_mode.c",1883,
2021-09-02 23:04:24.202702 CST,,,p16607,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","TransitionToMasterOrMirrorless: starting database",,,,,,,0,,"primary_mirror_mode.c",1897,
2021-09-02 23:04:24.204175 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","database system was shut down at 2021-09-02 23:04:14 CST",,,,,,,0,,"xlog.c",6371,
2021-09-02 23:04:24.204382 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","checkpoint record is at 33/BC0134B0",,,,,,,0,,"xlog.c",6559,
2021-09-02 23:04:24.204410 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","redo record is at 33/BC0134B0; undo record is at 0/0; shutdown TRUE",,,,,,,0,,"xlog.c",6616,
2021-09-02 23:04:24.204422 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","next transaction ID: 0/26107160; next OID: 91211021",,,,,,,0,,"xlog.c",6620,
2021-09-02 23:04:24.204430 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","next MultiXactId: 144516; next MultiXactOffset: 724843",,,,,,,0,,"xlog.c",6623,
2021-09-02 23:04:24.204466 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 33/BC013558",,,,,,,0,,"xlog.c",6986,
entstore.c",350,
entstore.c",350,
stentstore.c",350,
tentstore.c",350,
2021-09-02 23:04:24.227428 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","Oldest active transaction from prepared transactions 26107160",,,,,,,0,,"xlog.c",6033,
2021-09-02 23:04:24.248566 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",6084,
(GCC) 4.4.2 compiled on Jan 24 2017 20:25:07",,,,,,,0,,"xlog.c",6095,
2021-09-02 23:04:24.255618 CST,,,p16609,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","Finished normal startup for clean shutdown case",,,,,,,0,,"xlog.c",7216,
2021-09-02 23:04:24.283893 CST,,,p16610,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","Finished BASIC startup integrity checking",,,,,,,0,,"xlog.c",7661,
(GCC) 4.4.2 compiled on Jan 24 2017 20:27:17",,,,,,,0,,"postmaster.c",4618,
on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jan 24 2017 20:27:17",,,,,,0,,"postmaster.c",4624,
2021-09-02 23:04:24.289886 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","SNMP inform/trap alerts are disabled",,,,,,,,"send_snmp_inform_or_trap","sendalert.c",228,
2021-09-02 23:04:24.289920 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","e-mail alerts are disabled",,,,,,,,"send_alert_via_email","sendalert.c",569,
,,,
,,,,,0,,"ftsfilerep.c",358,
2021-09-02 23:04:24.303331 CST,,,p16615,th-1357596800,,,,0,con2,,seg-1,,,,,"LOG","00000","FTS: change state for segment (dbid=3, content=1) from ('u','p') to ('u','p')",,,,,,,0,,"fts.c",1157,
2021-09-02 23:04:24.303347 CST,,,p16615,th-1357596800,,,,0,con2,,seg-1,,,,,"LOG","00000","FTS: change state for segment (dbid=7, content=1) from ('u','m') to ('d','m')",,,,,,,0,,"fts.c",1157,
2021-09-02 23:04:24.303362 CST,,,p16615,th-1357596800,,,,0,con2,,seg-1,,,x26107160,sx1,"LOG","00000","probeUpdateConfig called for 2 changes",,,,,,,0,,"fts.c",976,
,,,,0,,"ftsfilerep.c",498,
2021-09-02 23:04:24.602216 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Success:",,,,,,,,"SysLoggerMain","syslogger.c",550,
ions:",,,,,,,0,,"cdbtm.c",3609,
ions:",,,,,,,0,,"cdbtm.c",3609,
sd1:40000",,,,,,,0,,"cdbgang.c",430,
seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
sd1:40001",,,,,,,0,,"cdbgang.c",430,
seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
set because its gang was disconnected (session id = 5). The new session id = 6",,,,,,,0,,"cdbgang.c",2685,
sd1:40000",,,,,,,0,,"cdbgang.c",430,
seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
sd1:40001",,,,,,,0,,"cdbgang.c",430,
seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
set because its gang was disconnected (session id = 6). The new session id = 7",,,,,,,0,,"cdbgang.c",2685,
sd1:40000",,,,,,,0,,"cdbgang.c",430,
seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
sd1:40001",,,,,,,0,,"cdbgang.c",430,
seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
set because its gang was disconnected (session id = 7). The new session id = 8",,,,,,,0,,"cdbgang.c",2685,
sd1:40000",,,,,,,0,,"cdbgang.c",430,
seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
sd1:40001",,,,,,,0,,"cdbgang.c",430,
seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
set because its gang was disconnected (session id = 8). The new session id = 9",,,,,,,0,,"cdbgang.c",2685,
sd1:40000",,,,,,,0,,"cdbgang.c",430,
seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
sd1:40001",,,,,,,0,,"cdbgang.c",430,
seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
eset because its gang was disconnected (session id = 9). The new session id = 10",,,,,,,0,,"cdbgang.c",2685,
allocated",,,,,,,0,,"cdbgang.c",1662,
exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
,,,,,,,0,,"cdbtm.c",1582,
tions:",,,,,,,0,,"cdbtm.c",3609,
tions:",,,,,,,0,,"cdbtm.c",3609,
psd1:40000",,,,,,,0,,"cdbgang.c",430,
o seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
psd1:40001",,,,,,,0,,"cdbgang.c",430,
o seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
eset because its gang was disconnected (session id = 10). The new session id = 11",,,,,,,0,,"cdbgang.c",2685,
psd1:40000",,,,,,,0,,"cdbgang.c",430,
o seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
psd1:40001",,,,,,,0,,"cdbgang.c",430,
o seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
eset because its gang was disconnected (session id = 11). The new session id = 12",,,,,,,0,,"cdbgang.c",2685,
psd1:40000",,,,,,,0,,"cdbgang.c",430,
o seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
psd1:40001",,,,,,,0,,"cdbgang.c",430,
o seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
eset because its gang was disconnected (session id = 12). The new session id = 13",,,,,,,0,,"cdbgang.c",2685,
psd1:40000",,,,,,,0,,"cdbgang.c",430,
o seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
psd1:40001",,,,,,,0,,"cdbgang.c",430,
o seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
eset because its gang was disconnected (session id = 13). The new session id = 14",,,,,,,0,,"cdbgang.c",2685,
psd1:40000",,,,,,,0,,"cdbgang.c",430,
o seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
psd1:40001",,,,,,,0,,"cdbgang.c",430,
o seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
eset because its gang was disconnected (session id = 14). The new session id = 15",,,,,,,0,,"cdbgang.c",2685,
allocated",,,,,,,0,,"cdbgang.c",1662,
exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
,,,,,,,0,,"cdbtm.c",1582,
on retry.",,,,,,,0,,"cdbtm.c",1602,
2021-09-02 23:04:25.742876 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /gp/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-22.base.el7.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
ure during startup recovery, retry failed, check segment status (cdbtm.c:1603)",,,,,,,0,,"cdbtm.c",1603,"Stack trace:
1 0xb06762 postgres <symbol not found> (elog.c:502)
2 0xb08778 postgres elog_finish (elog.c:1446)
3 0xcdaec2 postgres initTM (cdbtm.c:1497)
4 0xb1ec87 postgres InitPostgres (postinit.c:947)
5 0x99b04f postgres PostgresMain (postgres.c:4394)
6 0x8f955e postgres <symbol not found> (postmaster.c:6673)
7 0x8fc200 postgres PostmasterMain (postmaster.c:7604)
8 0x7fda6f postgres main (main.c:206)
9 0x7f4faaa2a3d5 libc.so.6 __libc_start_main + 0xf5
10 0x4bffe9 postgres <symbol not found> + 0x4bffe9
"
transactions:",,,,,,,0,,"cdbtm.c",3609,
transactions:",,,,,,,0,,"cdbtm.c",3609,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 16). The new session id = 17",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 17). The new session id = 18",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 18). The new session id = 19",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 19). The new session id = 20",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 20). The new session id = 21",,,,,,,0,,"cdbgang.c",2685,
er group allocated",,,,,,,0,,"cdbgang.c",1662,
, caught exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
l error",,,,,,,0,,"cdbtm.c",1582,
transactions:",,,,,,,0,,"cdbtm.c",3609,
transactions:",,,,,,,0,,"cdbtm.c",3609,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 21). The new session id = 22",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 22). The new session id = 23",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 23). The new session id = 24",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 24). The new session id = 25",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 25). The new session id = 26",,,,,,,0,,"cdbgang.c",2685,
er group allocated",,,,,,,0,,"cdbgang.c",1662,
, caught exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
l error",,,,,,,0,,"cdbtm.c",1582,
, failed on retry.",,,,,,,0,,"cdbtm.c",1602,
2021-09-02 23:04:27.747707 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /gp/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-22.base.el7.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
on: failure during startup recovery, retry failed, check segment status (cdbtm.c:1603)",,,,,,,0,,"cdbtm.c",1603,"Stack trace:
1 0xb06762 postgres <symbol not found> (elog.c:502)
2 0xb08778 postgres elog_finish (elog.c:1446)
3 0xcdaec2 postgres initTM (cdbtm.c:1497)
4 0xb1ec87 postgres InitPostgres (postinit.c:947)
5 0x99b04f postgres PostgresMain (postgres.c:4394)
6 0x8f955e postgres <symbol not found> (postmaster.c:6673)
7 0x8fc200 postgres PostmasterMain (postmaster.c:7604)
8 0x7fda6f postgres main (main.c:206)
9 0x7f4faaa2a3d5 libc.so.6 __libc_start_main + 0xf5
10 0x4bffe9 postgres <symbol not found> + 0x4bffe9
"
transactions:",,,,,,,0,,"cdbtm.c",3609,
transactions:",,,,,,,0,,"cdbtm.c",3609,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 27). The new session id = 28",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 28). The new session id = 29",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 29). The new session id = 30",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 30). The new session id = 31",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 31). The new session id = 32",,,,,,,0,,"cdbgang.c",2685,
er group allocated",,,,,,,0,,"cdbgang.c",1662,
, caught exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
l error",,,,,,,0,,"cdbtm.c",1582,
transactions:",,,,,,,0,,"cdbtm.c",3609,
transactions:",,,,,,,0,,"cdbtm.c",3609,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 32). The new session id = 33",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 33). The new session id = 34",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 34). The new session id = 35",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 35). The new session id = 36",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 36). The new session id = 37",,,,,,,0,,"cdbgang.c",2685,
er group allocated",,,,,,,0,,"cdbgang.c",1662,
, caught exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
l error",,,,,,,0,,"cdbtm.c",1582,
, failed on retry.",,,,,,,0,,"cdbtm.c",1602,
2021-09-02 23:04:33.456183 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /gp/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-22.base.el7.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
on: failure during startup recovery, retry failed, check segment status (cdbtm.c:1603)",,,,,,,0,,"cdbtm.c",1603,"Stack trace:
1 0xb06762 postgres <symbol not found> (elog.c:502)
2 0xb08778 postgres elog_finish (elog.c:1446)
3 0xcdaec2 postgres initTM (cdbtm.c:1497)
4 0xb1ec87 postgres InitPostgres (postinit.c:947)
5 0x99b04f postgres PostgresMain (postgres.c:4394)
6 0x8f955e postgres <symbol not found> (postmaster.c:6673)
7 0x8fc200 postgres PostmasterMain (postmaster.c:7604)
8 0x7fda6f postgres main (main.c:206)
9 0x7f4faaa2a3d5 libc.so.6 __libc_start_main + 0xf5
10 0x4bffe9 postgres <symbol not found> + 0x4bffe9
"
transactions:",,,,,,,0,,"cdbtm.c",3609,
transactions:",,,,,,,0,,"cdbtm.c",3609,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 38). The new session id = 39",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 39). The new session id = 40",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 40). The new session id = 41",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 41). The new session id = 42",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 42). The new session id = 43",,,,,,,0,,"cdbgang.c",2685,
er group allocated",,,,,,,0,,"cdbgang.c",1662,
, caught exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
l error",,,,,,,0,,"cdbtm.c",1582,
transactions:",,,,,,,0,,"cdbtm.c",3609,
transactions:",,,,,,,0,,"cdbtm.c",3609,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 43). The new session id = 44",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 44). The new session id = 45",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 45). The new session id = 46",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 46). The new session id = 47",,,,,,,0,,"cdbgang.c",2685,
to seg0 psd1:40000",,,,,,,0,,"cdbgang.c",430,
onnect to seg0 psd1:40000 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
to seg1 psd1:40001",,,,,,,0,,"cdbgang.c",430,
onnect to seg1 psd1:40001 with options FATAL: role 17147 does not exist",,,,,,,0,,"cdbgang.c",437,
on was reset because its gang was disconnected (session id = 47). The new session id = 48",,,,,,,0,,"cdbgang.c",2685,
er group allocated",,,,,,,0,,"cdbgang.c",1662,
, caught exception: looking for failed segments.",,,,,,,0,,"cdbtm.c",1577,
l error",,,,,,,0,,"cdbtm.c",1582,
, failed on retry.",,,,,,,0,,"cdbtm.c",1602,
2021-09-02 23:04:38.751381 CST,,,p16608,th-1357596800,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
addr2line: /gp/greenplum-db/./lib/libz.so.1: no version information available (required by /lib64/libbfd-2.25.1-22.base.el7.so)",,,,,,,,"SysLoggerMain","syslogger.c",550,
on: failure during startup recovery, retry failed, check segment status (cdbtm.c:1603)",,,,,,,0,,"cdbtm.c",1603,"Stack trace:
1 0xb06762 postgres <symbol not found> (elog.c:502)
2 0xb08778 postgres elog_finish (elog.c:1446)
3 0xcdaec2 postgres initTM (cdbtm.c:1497)
4 0xb1ec87 postgres InitPostgres (postinit.c:947)
5 0x99b04f postgres PostgresMain (postgres.c:4394)
6 0x8f955e postgres <symbol not found> (postmaster.c:6673)
7 0x8fc200 postgres PostmasterMain (postmaster.c:7604)
8 0x7fda6f postgres main (main.c:206)
9 0x7f4faaa2a3d5 libc.so.6 __libc_start_main + 0xf5
10 0x4bffe9 postgres <symbol not found> + 0x4bffe9
"
,"BEGIN",0,,"postgres.c",1619,
MIN_MESSAGES='ERROR'",,,,,,"SET CLIENT_MIN_MESSAGES='ERROR'",0,,"postgres.c",1619,
,,"COMMIT",0,,"postgres.c",1619,
,"BEGIN",0,,"postgres.c",1619,
ion()",,,,,,"SELECT version()",0,,"postgres.c",1619,
ame, typlen FROM pg_type WHERE oid=25",,,,,,"SELECT typname, typlen FROM pg_type WHERE oid=25",0,,"postgres.c",1619,
2021-09-02 23:04:40.902801 CST,"gpadmin","template1",p17466,th-1357596800,"::1","42569",2021-09-02 23:04:40 CST,26107168,con49,cmd9,seg-1,,,x26107168,sx1,"LOG","00000","statement:
SELECT fault_strategy FROM gp_fault_strategy",,,,,,"
SELECT fault_strategy FROM gp_fault_strategy
",0,,"postgres.c",1619,
... 单独登录各个节点查询
select oid,xmin,xmax,ctid,rolname,rolsuper from pg_authid;
oid | xmin | xmax | ctid | rolname | rolsuper
----------+----------+------+-------+-------------+----------
10 | 26102749 | 0 | (0,4) | gpadmin | f
(3 rows)select oid,xmin,xmax,ctid,rolname,rolsuper from pg_authid;
oid | xmin | xmax | ctid | rolname | rolsuper
----------+----------+------+-------+-------------+----------
10 | 26102749 | 0 | (0,5) | gpadmin |t
(3 rows) 发现各个的ctid,rolsuper并不一致
重新理解gprecoverseg
---
文档
在这一阶段，主Segment数据目录中的系统文件会被拷贝到该Segment的数据目录中。在下列平面文件被从主数据目录拷贝到该Segment数据目录的过程中，IO被暂停：
- pg_xlog/*
- pg_clog/*
- pg_distributedlog/*
- pg_distributedxidmap/*
- pg_multixact/members
- pg_multixact/offsets
- pg_twophase/*
- global/pg_database
- global/pg_auth
- global/pg_auth_time_constraint
这些文件被拷贝后，IOSUSPEND结束。
在Master上的容错服务器（ftsprobe）进程下一次苏醒时，它将把主Segment和镜像Segment的状态设置为同步（mode=s, state=u）。一次分布式查询也将会触发ftsprobe去更新该状态。
当所有Segment恢复和文件复制处理完成后，gp_segment_configuration系统表和gp_state -e的输出中的Segment状态如下表所示。
---
查看pg_atuhid的oid,xmin,xmax
发现各个节点的oid,xmin,xmax,rolsuper都不一样,确实说明一个问题就是gprecoverseg没有同步pg_authid没有同步
重新关闭gpstop -af
将主节点的master的四个文件拷贝到各个节点，尝试利用复制文件替换
$ gpstart -a 发现几个数据节点没有启动起来，暂时就先启动应用，后来重新执行gprecoverseg后，就彻底好了