数据库宕机信息
客户环境多套数据库宕机，error log信息如下：
2021-08-17T13:17:18.033537+08:00 4 [Note] Got an error reading communication packets
2021-08-17T13:17:43.281992+08:00 5 [Note] Got an error reading communication packets
2021-08-17T13:32:21.657931+08:00 8 [Note] Aborted connection 8 to db: 'test' user: 'root' host: '127.0.0.1' (Got an error reading communication packets)
2021-08-17T13:32:27.918403+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 5250ms. The settings might not be optimal. (flushed=1465 and evicted=0, during the time.)
2021-08-17T14:24:21.727815+08:00 12 [Note] Aborted connection 12 to db: 'recover' user: 'root' host: 'localhost' (Got timeout reading communication packets)
2021-08-17T14:43:19.642209+08:00 14 [Note] Got an error reading communication packets
2021-08-17T15:43:40.972887+08:00 26 [Note] Got an error reading communication packets
2021-08-17T17:02:02.057400+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 10947ms. The settings might not be optimal. (flushed=408 and evicted=0, during the time.)
2021-08-17T17:02:29.974975+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 4875ms. The settings might not be optimal. (flushed=328 and evicted=0, during the time.)
2021-08-17T17:03:44.284632+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 7881ms. The settings might not be optimal. (flushed=4000 and evicted=0, during the time.)
2021-08-17T17:04:07.271494+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 8948ms. The settings might not be optimal. (flushed=4000 and evicted=0, during the time.)
Status information:
Current dir: /data/mysql_data/
Running threads: 2 Stack size: 262144
Current locks:
lock: 0x7556b30:
lock: 0x74ecee0:
lock: 0x7494630:
lock: 0x74d6910:
lock: 0x74bdb70:
lock: 0x74b73f0:
lock: 0x74af850:
lock: 0x211f760:
lock: 0x211f680:
lock: 0x211f5a0:
lock: 0x211f4c0:
lock: 0x211f3e0:
lock: 0x211f300:
lock: 0x211f220:
lock: 0x211f140:
lock: 0x211f060:
lock: 0x211fd80:
lock: 0x211c560:
lock: 0x2120640:
lock: 0x2120560:
lock: 0x2120480:
lock: 0x21203a0:
lock: 0x21202c0:
lock: 0x21201e0:
lock: 0x2120100:
lock: 0x2120020:
lock: 0x211dce0:
lock: 0x211f840:
lock: 0x211e120:
lock: 0x211e040:
lock: 0x211df60:
lock: 0x211de80:
lock: 0x211ddc0:
lock: 0x211ff40:
lock: 0x211fe60:
lock: 0x211e580:
lock: 0x211e4a0:
lock: 0x211e3c0:
lock: 0x211e740:
lock: 0x211bca0:
lock: 0x211fca0:
lock: 0x211cc60:
lock: 0x211c9c0:
lock: 0x211cb80:
lock: 0x211c8e0:
lock: 0x211caa0:
lock: 0x211d200:
lock: 0x211d2c0:
lock: 0x211d380:
lock: 0x211c480:
lock: 0x211c3a0:
lock: 0x211c800:
lock: 0x211c2c0:
lock: 0x211c720:
lock: 0x211c1e0:
lock: 0x211c640:
lock: 0x211cfa0:
lock: 0x211d060:
lock: 0x211d120:
lock: 0x211c100:
lock: 0x211be60:
lock: 0x211c020:
lock: 0x211bd80:
lock: 0x211bf40:
lock: 0x211cd40:
lock: 0x211ce00:
lock: 0x211cec0:
lock: 0x211f920:
lock: 0x211fbc0:
lock: 0x211fae0:
lock: 0x211fa00:
lock: 0x211ed40:
lock: 0x211ec60:
lock: 0x211eb80:
lock: 0x211eaa0:
lock: 0x211e9c0:
lock: 0x211eec0:
lock: 0x211e900:
lock: 0x211e820:
lock: 0x211ef80:
lock: 0x211e660:
lock: 0x211e200:
lock: 0x211e2e0:
lock: 0x211dc00:
lock: 0x211db20:
lock: 0x211d7a0:
lock: 0x211da40:
lock: 0x211d960:
lock: 0x211d6c0:
lock: 0x211d880:
lock: 0x211d460:
lock: 0x211d520:
lock: 0x211d5e0:
lock: 0x211ee00:
Key caches:
default
Buffer_size: 8388608
Block_size: 1024
Division_limit: 100
Age_limit: 300
blocks used: 3
not flushed: 0
w_requests: 0
writes: 0
r_requests: 6
reads: 3
handler status:
read_key: 15
read_next: 5050002
read_rnd 0
read_first: 17
write: 5050100
delete 0
update: 0
Table status:
Opened tables: 123
Open tables: 108
Open files: 18
Open streams: 0
Memory status:
<malloc version="1">
<heap nr="0">
<sizes>
<size from="17" to="32" total="64" count="2"/>
<size from="33" to="48" total="192" count="4"/>
<size from="49" to="64" total="256" count="4"/>
<size from="65" to="80" total="80" count="1"/>
<size from="33" to="33" total="66" count="2"/>
<size from="49" to="49" total="49" count="1"/>
<size from="97" to="97" total="6596" count="68"/>
<size from="113" to="113" total="791" count="7"/>
<size from="145" to="145" total="290" count="2"/>
<size from="273" to="273" total="546" count="2"/>
<size from="353" to="353" total="1765" count="5"/>
<size from="449" to="449" total="449" count="1"/>
<size from="513" to="513" total="1026" count="2"/>
<size from="8241" to="8241" total="24723" count="3"/>
<size from="8753" to="8753" total="8753" count="1"/>
<size from="16369" to="16369" total="49107" count="3"/>
<size from="24529" to="24529" total="98116" count="4"/>
<size from="32689" to="32689" total="32689" count="1"/>
<size from="7729857" to="7729857" total="7729857" count="1"/>
</sizes>
<total type="fast" count="11" size="592"/>
<total type="rest" count="103" size="7954823"/>
<system type="current" size="103022592"/>
<system type="max" size="103022592"/>
<aspace type="total" size="103022592"/>
<aspace type="mprotect" size="103022592"/>
</heap>
<heap nr="1">
<sizes>
<size from="33" to="33" total="132" count="4"/>
<size from="81" to="81" total="81" count="1"/>
<size from="145" to="145" total="145" count="1"/>
<size from="161" to="161" total="161" count="1"/>
<size from="273" to="273" total="273" count="1"/>
<size from="353" to="353" total="353" count="1"/>
<size from="417" to="417" total="417" count="1"/>
<size from="433" to="433" total="433" count="1"/>
<size from="449" to="449" total="449" count="1"/>
<size from="465" to="465" total="465" count="1"/>
<size from="881" to="881" total="881" count="1"/>
<size from="897" to="897" total="897" count="1"/>
<size from="1361" to="1361" total="1361" count="1"/>
<size from="2241" to="2257" total="11221" count="5"/>
<unsorted from="897" to="66225" total="2507439" count="303"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="324" size="2524708"/>
<system type="current" size="339668992"/>
<system type="max" size="339668992"/>
<aspace type="total" size="4124672"/>
<aspace type="mprotect" size="4124672"/>
</heap>
<heap nr="2">
<sizes>
<size from="81" to="96" total="96" count="1"/>
<size from="33" to="33" total="297" count="9"/>
<size from="49" to="49" total="147" count="3"/>
<size from="65" to="65" total="260" count="4"/>
<size from="129" to="129" total="129" count="1"/>
<size from="145" to="145" total="145" count="1"/>
<size from="481" to="481" total="481" count="1"/>
<size from="513" to="513" total="513" count="1"/>
<size from="545" to="545" total="545" count="1"/>
<size from="561" to="561" total="561" count="1"/>
<size from="577" to="577" total="1154" count="2"/>
<size from="625" to="625" total="625" count="1"/>
<size from="689" to="689" total="689" count="1"/>
<size from="737" to="737" total="737" count="1"/>
<size from="769" to="769" total="769" count="1"/>
<size from="849" to="849" total="849" count="1"/>
<size from="945" to="945" total="945" count="1"/>
<size from="6257" to="6257" total="6257" count="1"/>
<size from="55489" to="55489" total="55489" count="1"/>
<unsorted from="33" to="33" total="33" count="1"/>
</sizes>
<total type="fast" count="1" size="96"/>
<total type="rest" count="33" size="70625"/>
<system type="current" size="3301376"/>
<system type="max" size="3301376"/>
<aspace type="total" size="3301376"/>
<aspace type="mprotect" size="3301376"/>
</heap>
<heap nr="3">
<sizes>
<size from="33" to="48" total="240" count="5"/>
<size from="49" to="64" total="64" count="1"/>
<size from="65" to="80" total="240" count="3"/>
<size from="33" to="33" total="66" count="2"/>
<size from="65" to="65" total="65" count="1"/>
<size from="241" to="241" total="482" count="2"/>
<size from="273" to="273" total="273" count="1"/>
<size from="625" to="625" total="625" count="1"/>
<size from="1009" to="1009" total="1009" count="1"/>
<size from="1025" to="1025" total="1025" count="1"/>
<size from="1249" to="1249" total="1249" count="1"/>
<size from="1313" to="1313" total="1313" count="1"/>
<size from="1585" to="1585" total="1585" count="1"/>
<size from="2721" to="2721" total="2721" count="1"/>
<size from="16417" to="16417" total="16417" count="1"/>
</sizes>
<total type="fast" count="9" size="544"/>
<total type="rest" count="14" size="26830"/>
<system type="current" size="151552"/>
<system type="max" size="151552"/>
<aspace type="total" size="151552"/>
<aspace type="mprotect" size="151552"/>
</heap>
<heap nr="4">
<sizes>
<size from="65" to="80" total="80" count="1"/>
<size from="81" to="81" total="81" count="1"/>
<size from="113" to="113" total="113" count="1"/>
<size from="241" to="241" total="241" count="1"/>
</sizes>
<total type="fast" count="1" size="80"/>
<total type="rest" count="3" size="435"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="5">
<sizes>
<size from="113" to="113" total="113" count="1"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="1" size="113"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="6">
<sizes>
<size from="113" to="113" total="113" count="1"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="1" size="113"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="7">
<sizes>
<size from="65" to="65" total="65" count="1"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="1" size="65"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="8">
<sizes>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="0" size="0"/>
<system type="current" size="389120"/>
<system type="max" size="389120"/>
<aspace type="total" size="389120"/>
<aspace type="mprotect" size="389120"/>
</heap>
<total type="fast" count="22" size="1312"/>
<total type="rest" count="480" size="10577712"/>
<total type="mmap" count="81" size="362438656"/>
<system type="current" size="447074304"/>
<system type="max" size="447074304"/>
<aspace type="total" size="111529984"/>
<aspace type="mprotect" size="111529984"/>
</malloc>
Events status:
LLA = Last Locked At LUA = Last Unlocked At
WOC = Waiting On Condition DL = Data Locked
Event scheduler status:
State : INITIALIZED
Thread id : 0
LLA : n/a:0
LUA : n/a:0
WOC : NO
Workers : 0
Executed : 0
Data locked: NO
Event queue status:
Element count : 0
Data locked : NO
Attempting lock : NO
LLA : init_queue:103
LUA : init_queue:111
WOC : NO
Next activation : never
2021-08-17T17:08:10.123540+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 8773ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
2021-08-17T17:08:27.584420+08:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 7896ms. The settings might not be optimal. (flushed=3417 and evicted=0, during the time.)
2021-08-17T17:08:27.871717+08:00 32 [Note] Aborted connection 32 to db: 'test' user: 'root' host: '127.0.0.1' (Got an error reading communication packets)
2021-08-17T17:43:32.012491+08:00 34 [Note] Got an error reading communication packets
2021-08-17T17:45:25.898084+08:00 35 [Note] Got packets out of order
2050-10-10T20:00:05.358937+08:00 36 [Warning] Current time has got past year 2038. Validating current time with 5 iterations before initiating the normal server shutdown process.
2050-10-10T20:00:05.385232+08:00 36 [Warning] Iteration 1: Current time obtained from system is greater than 2038
2050-10-10T20:00:05.385246+08:00 36 [Warning] Iteration 2: Current time obtained from system is greater than 2038
2050-10-10T20:00:05.385250+08:00 36 [Warning] Iteration 3: Current time obtained from system is greater than 2038
2050-10-10T20:00:05.385253+08:00 36 [Warning] Iteration 4: Current time obtained from system is greater than 2038
2050-10-10T20:00:05.385256+08:00 36 [Warning] Iteration 5: Current time obtained from system is greater than 2038
2050-10-10T20:00:05.408669+08:00 36 [ERROR] This MySQL server doesn't support dates later than 2038
2050-10-10T20:00:05.746302+08:00 0 [Note] Giving 0 client threads a chance to die gracefully
2050-10-10T20:00:05.746331+08:00 0 [Note] Shutting down slave threads
2050-10-10T20:00:05.746340+08:00 0 [Note] Forcefully disconnecting 0 remaining clients
2050-10-10T20:00:05.770946+08:00 0 [Note] Event Scheduler: Purging the queue. 0 events
2050-10-10T20:00:05.822777+08:00 0 [Note] Binlog end
2050-10-10T20:00:05.942614+08:00 0 [Note] Shutting down plugin 'ngram'
2050-10-10T20:00:05.948141+08:00 0 [Note] Shutting down plugin 'partition'
2050-10-10T20:00:05.948176+08:00 0 [Note] Shutting down plugin 'BLACKHOLE'
2050-10-10T20:00:05.948187+08:00 0 [Note] Shutting down plugin 'ARCHIVE'
2050-10-10T20:00:05.948192+08:00 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2050-10-10T20:00:05.964149+08:00 0 [Note] Shutting down plugin 'MRG_MYISAM'
2050-10-10T20:00:05.964186+08:00 0 [Note] Shutting down plugin 'MyISAM'
2050-10-10T20:00:05.967164+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2050-10-10T20:00:05.968176+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2050-10-10T20:00:05.968196+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2050-10-10T20:00:05.968200+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2050-10-10T20:00:05.968204+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2050-10-10T20:00:05.968207+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2050-10-10T20:00:05.968210+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2050-10-10T20:00:05.968214+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2050-10-10T20:00:05.968217+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2050-10-10T20:00:05.968220+08:00 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2050-10-10T20:00:05.968223+08:00 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2050-10-10T20:00:05.968226+08:00 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2050-10-10T20:00:05.968229+08:00 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2050-10-10T20:00:05.968232+08:00 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2050-10-10T20:00:05.968235+08:00 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2050-10-10T20:00:05.972522+08:00 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2050-10-10T20:00:05.972537+08:00 0 [Note] Shutting down plugin 'INNODB_METRICS'
2050-10-10T20:00:05.972545+08:00 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2050-10-10T20:00:05.972549+08:00 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2050-10-10T20:00:05.972554+08:00 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2050-10-10T20:00:05.972558+08:00 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2050-10-10T20:00:05.972562+08:00 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2050-10-10T20:00:05.972568+08:00 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2050-10-10T20:00:05.972576+08:00 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2050-10-10T20:00:05.972580+08:00 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2050-10-10T20:00:05.972584+08:00 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2050-10-10T20:00:05.972588+08:00 0 [Note] Shutting down plugin 'INNODB_CMP'
2050-10-10T20:00:05.972593+08:00 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2050-10-10T20:00:05.972597+08:00 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2050-10-10T20:00:05.972601+08:00 0 [Note] Shutting down plugin 'INNODB_TRX'
2050-10-10T20:00:05.972606+08:00 0 [Note] Shutting down plugin 'InnoDB'
2050-10-10T20:00:06.009896+08:00 0 [Note] InnoDB: FTS optimize thread exiting.
2050-10-10T20:00:06.010211+08:00 0 [Note] InnoDB: Starting shutdown...
2050-10-10T20:00:06.111955+08:00 0 [Note] InnoDB: Dumping buffer pool(s) to /data/mysql_data/ib_buffer_pool
2050-10-10T20:00:06.135351+08:00 0 [Note] InnoDB: Buffer pool(s) dump completed at 501010 20:00:06
2050-10-10T20:00:27.872226+08:00 0 [Note] InnoDB: Shutdown completed; log sequence number 3757316759
2050-10-10T20:00:27.881295+08:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2050-10-10T20:00:27.881333+08:00 0 [Note] Shutting down plugin 'MEMORY'
2050-10-10T20:00:27.883896+08:00 0 [Note] Shutting down plugin 'CSV'
2050-10-10T20:00:27.883923+08:00 0 [Note] Shutting down plugin 'sha256_password'
2050-10-10T20:00:27.883934+08:00 0 [Note] Shutting down plugin 'mysql_native_password'
2050-10-10T20:00:27.907348+08:00 0 [Note] Shutting down plugin 'binlog'
2050-10-10T20:00:27.945216+08:00 0 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete[ERROR] This MySQL server doesn't support dates later than 2038
从上述错误可以看出，超过MySQL支持的时间范围，经过确认，宕机的数据库都为vmware上的虚拟机，从下面可以看出，虚拟机时间取自于宿主机，运维人员误更改宿主机系统时间导致
那么我们来看下为什么时间不能超过2038年呢？
千年虫2038年问题
和21世纪初的千年虫（the Millennium bug）问题类似，32位的Unix操作系统和Linux操作系统时间溢出问题又称为2038年问题（the Year 2038 problem）。如果你想知道什么是2038问题的话，你需要知道一些技术上的东西。这个bug是由用来写Unix/Linux的C语言引起的，C语言中用 time_t 来代表时间和日期，time_t 是整数（int）型的，它用来记载从1970年1月1日到2000年所经历的秒数。
这个数据是以32位存储的，第一位是符号位，其余的31位用来存数字，而这31位数字可以存储的最大数字为2147483647。
从1970年开始计算，这31位的数字可以表示的秒数最多可以用到2038年01月19日03时14分07秒，当时间到达这个数字的时候系统将会出现问题，到时候数字不会自动增加，而是会变为-2147483648，而这串数字代表的时间是1901年12月13日20时45分52秒，这会导致很多的程序出现问题，甚至崩溃。
2038年问题不仅比千年虫更隐蔽，而且比之前千年虫问题更具有破坏力，因为千年虫问题只会导致应用层的程序出现问题，比如信用卡支付系统，或者管理系统。而2038这个bug，将会影响系统最底层的时间控制的功能。
要解决这个问题，最简单的方式是扩展Unix时间的长度，用64位数字来表示它。64位二进制数的实际可用位数是63位，最大表示到公历的UTC时间292,277,026,596年12月4日15时30分08秒. 如果那个时候人类文明还存在的话，公元纪年很可能已经因为太难用而被抛弃了. 理想的情况是到2038年，64位系统已经成为主流，从而避免特意去修正这个问题所需要的大量开销。否则，人们就必须把新的64位时间拆分成两部分并分别保存在两个变量里，这是一个麻烦而且效率低下的选择.
8.0.26版本仍然有这个问题
if (thd->killed == THD::KILL_QUERY) thd->killed = THD::NOT_KILLED;
thd->set_time();
if (is_time_t_valid_for_timestamp(thd->query_start_in_secs()) == false) {
/*
If the time has gone past 2038 we need to shutdown the server. But
there is possibility of getting invalid time value on some platforms.
For example, gettimeofday() might return incorrect value on solaris
platform. Hence validating the current time with 5 iterations before
initiating the normal server shutdown process because of time getting
past 2038.
*/
const int max_tries = 5;
LogErr(WARNING_LEVEL, ER_CONFIRMING_THE_FUTURE, max_tries);
int tries = 0;
while (++tries <= max_tries) {
thd->set_time();
if (is_time_t_valid_for_timestamp(thd->query_start_in_secs()) == true) {
LogErr(WARNING_LEVEL, ER_BACK_IN_TIME, tries);
break;
}
LogErr(WARNING_LEVEL, ER_FUTURE_DATE, tries);
}
if (tries > max_tries) {
/*
If the time has got past 2038 we need to shut this server down
We do this by making sure every command is a shutdown and we
have enough privileges to shut the server down
TODO: remove this when we have full 64 bit my_time_t support
*/从以上这两段注释来看， MySQL 内部变量 my_time_t 不支持 64 位，在到达2038年之后，MySQL 无法执行任何命令，直接关闭。
从mysql_time.h中可以看到TEMESTAMP时间最大值定义为int32
constexpr const int TIMESTAMP_MAX_VALUE =
std::numeric_limits<std::int32_t>::max();
constexpr const int TIMESTAMP_MIN_VALUE = 1;时间字段选择
在MySQL中时间字段，timestamp最大值也为2038年，这也是由于timestamp实际存储的是int值，
可以看到在一些开发规范中也是要使用datetime类型。
链接文字