上一篇介绍了因为子表过多,导致innodb crash的情况,但crash的原因是long semaphore waits。
long semaphore waits又为何物?
背景:Innodb使用了mutex和rw_lock来保护内存数据结构,同步的机制要么是互斥,要么是读写阻塞的模式,
Innode认为mutex和rw_lock hold的时间足够短,所以,如果有线程wait mutex或者rw_lock时间过长,那么
很可能是程序有bug,所以就会异常主动crash。
关于innodb的锁:可以参考前面介绍的blog。
1. Innodb如何界定时间过长?
/* The following is the maximum allowed duration of a lock wait. */ UNIV_INTERN ulint srv_fatal_semaphore_wait_threshold = 600;
Innodb的认为600s足够长。
2. Innodb如何判断?
/* Create the thread which warns of long semaphore waits */ os_thread_create(&srv_error_monitor_thread, NULL, thread_ids + 3 + SRV_MAX_N_IO_THREADS);
innodb后台启动了一个线程,专门监控wait队列,
if (sync_array_print_long_waits(&waiter, &sema) && sema == old_sema && os_thread_eq(waiter, old_waiter)) { fatal_cnt++; if (fatal_cnt > 10) { fprintf(stderr, "InnoDB: Error: semaphore wait has lasted" " > %lu seconds\n" "InnoDB: We intentionally crash the server," " because it appears to be hung.\n", (ulong) srv_fatal_semaphore_wait_threshold); ut_error; }
判断10次,如果是同一个线程,等待同一个sema,那么就crash掉Innodb。
for (i = 0; i < sync_primary_wait_array->n_cells; i++) { double diff; void* wait_object; cell = sync_array_get_nth_cell(sync_primary_wait_array, i); wait_object = cell->wait_object; if (wait_object == NULL || !cell->waiting) { continue; } diff = difftime(time(NULL), cell->reservation_time); if (diff > SYNC_ARRAY_TIMEOUT) { fputs("InnoDB: Warning: a long semaphore wait:\n", stderr); sync_array_cell_print(stderr, cell); noticed = TRUE; } if (diff > fatal_timeout) { fatal = TRUE; } if (diff > longest_diff) { longest_diff = diff; *sema = wait_object; *waiter = cell->thread;
sync_primary_wait_array是一个数组,每一个wait sema的进入队列, 找到等待时间最长的,并且大于600s的,就设置fatal=TRUE。