mardi 21 avril 2015

how to run scalding test in local mode with local input file

Scalding has a great utility to run an integration test for the job flow. In this way the inputs and outputs are the in-memory buffer

val input = List("0" -> "This a a day")
val expectedOutput = List(("This", 1),("a", 2),("day", 1))
 JobTest(classOf[WordCountJob].getName)
  .arg("input", "input-data")
  .arg("output", "output-data")
  .source(TextLine("input-data"), input)
  .sink(Tsv("output-data")) {
  buffer: mutable.Buffer[(String, Int)] => {
    buffer should equal(expectedOutput)
  }
}.run

How can I transfare/write another code that will read input and write output to the real local file? Like FileTap/LFS in cascading - and not an in-memory approach

Aucun commentaire:

Enregistrer un commentaire