Spark主备切换机制原理

Master实际上可以配置两个,那么在spark原生的standalone上也是支持Master主备切换的,也就是说,当Active Master节点挂掉之后,我们可以将Standby Master切换为Active Master

Spark Master的主备切换可以基于两种切换机制,一种是文件系统,一种是基于Zookeeper,基于文件系统的机制,是Active Master挂掉后,需要我们手动去切换到Standby Master上,基于Zookeeper机制,呆以实现自动切换。

所以这里说的主备切换机制,其实指的是在Active Master挂掉之后,切换到Standby Master时,Master会做哪些操作

1.使用持久化引挚(FileSystemPersistence或者是ZookeeperPersisitence)去读取持久化的storedApps,storedDriver,storedWorker,

2.判断上面的三个持久化的storedApps,storedDriver,storedWorker,

有任何一个不为空,就将持久化有Application,Driver,Worker的信息重新注册,注册到Master内部的缓存结构中。

3.将Application和Worker的状态都修改为UNKNOWN,然后向Application对应的Driver,Worker发送Standby Master的地址.

4.Driver,Worker,理论上讲,如果他们目前是正常工作的话,那么在收到Master发送来的地址后,就会返回响应给新的Master。

5.此时,Master在陆续接收到Driver,Worker发送来的响应消息之后,会使用completeRecovery()对没有收到发送响应消息的Driver,Worker进行处理,过滤掉他们的信息。如下:

// Kill off any workers and apps that didn’t respond to us.
workers.filter(.state == WorkerState.UNKNOWN).foreach(removeWorker)
apps.filter(
.state == ApplicationState.UNKNOWN).foreach(finishApplication)

  1.   // Reschedule drivers which were not claimed by any workers
  2.   drivers.filter(_.worker.isEmpty).foreach { d =>
  3.   logWarning(s"Driver ${d.id} was not found after master recovery")
  4.   if (d.desc.supervise) {
  5.   logWarning(s"Re-launching ${d.id}")
  6.   relaunchDriver(d)
  7.   } else {
  8.   removeDriver(d.id, DriverState.ERROR, None)
  9.   logWarning(s"Did not re-launch ${d.id} because it was not supervised")
  10.   }

6.调用Master的schedule(),对正在等待调度的Driver,Application进行调度,比如在某个Worker上启动Driver,或者为Application在Worker上启动Executor。

state = RecoveryState.ALIVE
schedule()
了解更多大数据面试问题欢迎关注小编大数据培训专栏!

上一篇:An*id的IPC机制-Binder概述


下一篇:fiddler不能抓某些的包的原因