Accumulators

copy的官方programming guide的代码,只有一个要说一下,Accumulators的更新是在action做的,transformation操作中对于Accumulators的增加不会即时更新的,只有action只会更新。

scala> val accum = sc.accumulator(0, "My Accumulator")
accum: spark.Accumulator[Int] = 0 scala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)
...
10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s scala> accum.value
res2: Int = 10 object VectorAccumulatorParam extends AccumulatorParam[Vector] {
def zero(initialValue: Vector): Vector = {
Vector.zeros(initialValue.size)
}
def addInPlace(v1: Vector, v2: Vector): Vector = {
v1 += v2
}
} // Then, create an Accumulator of this type:
val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam)

  

上一篇:Qt 内存管理机制


下一篇:Gatling实战(一)