"If you want to run dataflow job using scala directly, then it will not be possible, because dataflow doesn't support Scala directly. To run Scala code in Dataflow, you must use Scio, a Scala API developed by Spotify that builds on top of the open-source Apache Beam SDK. Since Dataflow executes pipelines written using the Apache Beam model, Scio provides the necessary functionality to define and execute your Scala pipelines on the Dataflow runner."
"(i) Scala Build Tool (sbt): For a Scala project, sbt is the standard build tool used to compile your code and manage dependencies. (ii) Add Scio dependency: Include the Scio library and the appropriate Beam artifact in your build.sbt file. libraryDependencies += "com.spotify" %% "scio-core" % "..." // Use latest versionlibraryDependencies += "org.apache.beam" % "beam-runners-google-cloud-dataflow-java" % "..." // Use latest version"
Dataflow does not support Scala directly, so Scio, a Scala API from Spotify built on Apache Beam, is required to run Scala pipelines on the Dataflow runner. A Scala project requires sbt as the build tool and must add Scio and the appropriate Beam Dataflow runner artifacts to build.sbt. A Google Cloud project with billing enabled and the Dataflow API must be configured, and authentication should be performed via the gcloud CLI. Scio pipelines follow the Apache Beam programming model using idiomatic Scala. Example code includes defining custom pipeline options, creating a Scio context via ContextAndArgs, and implementing transforms like word count.
Read at Medium
Unable to calculate read time
Collection
[
|
...
]