每个job被划分为多个stage。划分stage的一个主要依据是当前计算因子的输入是否是确定的，如果是则将其分在同一个stage，从而避免多个stage之间的消息传递开销。

http://spark.apache.org/docs/latest/rdd-programming-guide.html

【Spark actions are executed through a set of stages, separated by distributed “shuffle” operations. 】

Shared Variables

Normally, when a function passed to a Spark operation (such as map or reduce) is executed on a remote cluster node, it works on separate copies of all the variables used in the function. These variables are copied to each machine, and no updates to the variables on the remote machine are propagated back to the driver program. Supporting general, read-write shared variables across tasks would be inefficient. However, Spark does provide two limited types of shared variables for two common usage patterns: broadcast variables and accumulators.

Broadcast Variables

Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used, for example, to give every node a copy of a large input dataset in an efficient manner. Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost.

Spark actions are executed through a set of stages, separated by distributed “shuffle” operations. Spark automatically broadcasts the common data needed by tasks within each stage. The data broadcasted this way is cached in serialized form and deserialized before running each task. This means that explicitly creating broadcast variables is only useful when tasks across multiple stages need the same data or when caching the data in deserialized form is important.

Broadcast variables are created from a variable v by calling SparkContext.broadcast(v). The broadcast variable is a wrapper around v, and its value can be accessed by calling the value method. The code below shows this:

• Scala
• Java
• Python

scala> val broadcastVar = sc.broadcast(Array(1, 2, 3))

broadcastVar: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)

scala> broadcastVar.value

res0: Array[Int] = Array(1, 2, 3)

After the broadcast variable is created, it should be used instead of the value v in any functions run on the cluster so that v is not shipped to the nodes more than once. In addition, the object v should not be modified after it is broadcast in order to ensure that all nodes get the same value of the broadcast variable (e.g. if the variable is shipped to a new node later).

Accumulators

Accumulators are variables that are only “added” to through an associative and commutative operation and can therefore be efficiently supported in parallel. They can be used to implement counters (as in MapReduce) or sums. Spark natively supports accumulators of numeric types, and programmers can add support for new types.

As a user, you can create named or unnamed accumulators. As seen in the image below, a named accumulator (in this instance counter) will display in the web UI for the stage that modifies that accumulator. Spark displays the value for each accumulator modified by a task in the “Tasks” table.

Tracking accumulators in the UI can be useful for understanding the progress of running stages (NOTE: this is not yet supported in Python).

• Scala
• Java
• Python

scala> val accum = sc.longAccumulator("My Accumulator")

accum: org.apache.spark.util.LongAccumulator = LongAccumulator(id: 0, name: Some(My Accumulator), value: 0)

scala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum.add(x))

...

10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s

scala> accum.value

res2: Long = 10

class VectorAccumulatorV2 extends AccumulatorV2[MyVector, MyVector] {

  private val myVector: MyVector = MyVector.createZeroVector

  def reset(): Unit = {

    myVector.reset()

  }

  def add(v: MyVector): Unit = {

    myVector.add(v)

  }

  ...

}

// Then, create an Accumulator of this type:

val myVectorAcc = new VectorAccumulatorV2

// Then, register it into spark context:

sc.register(myVectorAcc, "MyVectorAcc1")

For accumulator updates performed inside actions only, Spark guarantees that each task’s update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task’s update may be applied more than once if tasks or job stages are re-executed.

Accumulators do not change the lazy evaluation model of Spark. If they are being updated within an operation on an RDD, their value is only updated once that RDD is computed as part of an action. Consequently, accumulator updates are not guaranteed to be executed when made within a lazy transformation like map(). The below code fragment demonstrates this property:

• Scala
• Java
•  Python

Scala
Java
Python
val accum = sc.longAccumulator
data.map { x => accum.add(x); x }
// Here, accum is still 0 because no actions have caused the map operation to be computed.

Narrow Wide Dependencies 宽依赖 窄依赖

shuffle

窄依赖 宽依赖 解析

Wide vs Narrow Dependencies · rohgar/scala-spark-4 Wiki · GitHub https://github.com/rohgar/scala-spark-4/wiki/Wide-vs-Narrow-Dependencies

窄 父RDD的一个分区，在RDD转换过程中，最多只会被一个子RDD的分区使用

宽 多个

一个Spark应用程序会被分解为多个作业Job提交给Spark集群进行处理。然而，作业并不是应用程序被执行的最小计算单元。

Spark集群收到作业后还要就爱那个作业继续进行2次切分和规划，并进行相应的调度。

第一步：将作业按照RDD转换操作处理为最小的处理单元，即任务Task

第二步：对任务进行规划，生成包含多个任务的阶段Stage

这两步都是由SparkContext创建的DAGScheduler实例进行

其输入是一个作业中的RDD有向无环图，输出为一系列任务

对应宽依赖，应尽量切分到不同的阶段中，以避免过大的网络传输和计算开销

连续窄依赖的RDD转换，则尽量多地放入同一个阶段

一旦遇到一个宽依赖的类型的RDD转换操作，则生成一个新的阶段

整个依赖链被划分为多个stage阶段，每个stage内都是一组互相关联、但彼此之间没有shuffle依赖关系的任务集合，称为任务集 Taskset。

DAGScheduler不仅负责将作业分割为多个阶段，还负责管理调度阶段的提交。

waitingStages集合

runningStages集合

failedStages集合

DAGScheduler是反向遍历整个RDD依赖链

Task调度

DAG

DAGScheduler

TaskScheduler 为每一个收到的TaskSet创建一个TaskSetManager，TaskSetManager决定每个Task应该在哪个物理在哪个物理资源上执行，并将调度计划发给TaskScheduler，TaskScheduler将Task提交给Spark集群实际执行。

TaskScheduler通过ShedulerBackend来了解可以分给应用程序的物理资源的的情况。

TaskSetManager不与底层实际的物理节点通信。

Scheduling Within an Application

Job Scheduling - Spark 2.3.0 Documentation http://spark.apache.org/docs/latest/job-scheduling.html

作业的两种调度顺序