HA模式强制手动切换：IPC's epoch [X] is less than the last promised epoch [X+1]

2022-10-29 07:39:38

-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal 192.168.58.183: failed to write txns -. Will try to write to this JN again after the next log roll.

    at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:)

    at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:)

    at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$.callBlockingMethod(QJournalProtocolProtos.java:)

    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

    at org.apache.hadoop.ipc.Client.call(Client.java:)

    at org.apache.hadoop.ipc.Client.call(Client.java:)

    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:)

    at com.sun.proxy.$Proxy10.journal(Unknown Source)

    at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.journal(QJournalProtocolTranslatorPB.java:)

    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$.call(IPCLoggerChannel.java:)

    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$.call(IPCLoggerChannel.java:)

    at java.util.concurrent.FutureTask.run(FutureTask.java:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal 192.168.58.181: failed to write txns -. Will try to write to this JN again after the next log roll.

org.apache.hadoop.ipc.RemoteException(java.io.IOException): IPC's epoch 11 is less than the last promised epoch 12

    at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:)

    at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.journal(QJournalProtocolServerSideTranslatorPB.java:)

    at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$.callBlockingMethod(QJournalProtocolProtos.java:)

    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:)

    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at org.apache.hadoop.ipc.Server$Handler$.run(Server.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:)

    at org.apache.hadoop.ipc.Client.call(Client.java:)

    at org.apache.hadoop.ipc.Client.call(Client.java:)

    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:)

    at com.sun.proxy.$Proxy10.journal(Unknown Source)

    at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolTranslatorPB.journal(QJournalProtocolTranslatorPB.java:)

    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$.call(IPCLoggerChannel.java:)

    at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$.call(IPCLoggerChannel.java:)

    at java.util.concurrent.FutureTask.run(FutureTask.java:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

-- ::, WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Remote journal 192.168.58.182: failed to write txns -. Will try to write to this JN again after the next log roll.

org.apache.hadoop.ipc.RemoteException(java.io.IOException): IPC's epoch 11 is less than the last promised epoch 12

    at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:)

    at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:)

一、错误起因

　　Active NameNode日志出现异常IPC's epoch [X] is less than the last promised epoch [X+1]，出现短期的双Active

　　我配置的ha自动切换，但是发现STandByNameNode是active，我强制手动切换了三次，STandByNameNode就无法访问了，估计是这个问题。

二.内部原因

　　【HDFS机制】：该问题属于hdfs对于脑列的异常保护，属于正常行为，不影响业务。

　　1）ZKFC1对NameNode1(Active)进行健康检查，因为长时间监控不到NN1的回复，认为该NameNode1不健康，主动释放zk中的ActiveStandbyElectorLock，此时NN1还是active（因为zkfc与NameNode1连接异常，不能将其shutdown）。

　　

zkfc log：

　　-- ::, WARN org.apache.hadoop.ha.HealthMonitor: Transport-level exception trying to monitor health of NameNode at namenode01/172.21.248.14:: Call From namenode01/

　　72.21.248.14 to namenode02: failed on socket timeout exception: java.net.SocketTimeoutException:  millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[co

nnected local=/172.21.248.14: remote=namenode01/172.21.248.14:]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout

-- ::, WARN org.apache.hadoop.ha.FailoverController: Unable to gracefully make NameNode at namenode02/172.21.248.13: standby (unable to connect)

java.net.SocketTimeoutException: Call From namenode01/172.21.248.14 to namenode02: failed on socket timeout exception: java.net.SocketTimeoutException:  millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.21.248.14: remote=namenode02/172.21.248.13:]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout

　　2）ZKFC2在zk中竞争到ActiveStandbyElectorLock，将NameNode2（原来的Standby）变成Active，同时会更新JN中的epoch使其+1。

　　3）NameNode1（原先的Active）再次去操作JournalNode的editlog时发现自己的epoch比JN的epoch小1，促使自己重启，成为Standby NameNode。

NN1 log:

-- ::, FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [10.1.1.107:, 192.10.1.208:,

192.10.1.209:], stream=QuorumOutputStream starting at txid )) org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size /.  exceptions thrown: 192.10.1.208:: IPC's epoch 115 is less than the last promised epoch 116

三.解决方案

　　可以在core-site.xml文件中修改ha.health-monitor.rpc-timeout.ms参数值，来扩大zkfc监控检查超时时间。

<property>

<name>ha.health-monitor.rpc-timeout.ms</name>

<value>180000</value>

</property>

　　

四、结束语

　　最后设置成手动切换吧...其实可以通过zookeeper来找到那个是active，我先不这么做吧。在hdfs-site.xml。

　　但是设置成不自动切换的话，zkfc就取法启动，hbase必须用自己的zookeeper。

上一篇：AM335x tscadc platform driver 相关代码跟踪


下一篇：第11课 Qt中的字符串类