mardi 21 avril 2015

Task not serializable when using object in REPL

So, another SO question prompted me to try the following:

object Foo{
  def f = 1
}

sc.parallelize(List(1)).map(x=>{
  val myF = Foo.f _
  x + myF()
}

Which works, but the following does not:

object Foo{
  def f = 1
  def run(rdd: org.apache.spark.rdd.RDD[Int]) = rdd.map(x=>{
    val myF = Foo.f _
    x + myF()
  }
}

Foo.run(sc.parallelize(List(1)))

I will take a deeper look at the serialization stack tomorrow when I can remove the mess of the REPL output. But, this should be the same thing. Why does one way yell and the other does not.

Aucun commentaire:

Enregistrer un commentaire