TPC-DS on Spark 4.0+ : A Practical Guide and Benchmarking Considerations

"The spark-sql-perf toolkit doesn't work for Spark 4.0+ currently, and this guide shows you how to get it running(with a custom patch). While many developers have their own complex Spark setup, this workflow is designed to be simple and reproducible. It only requires an AWS account to provision a cluster and run a full benchmark from scratch. We'll focus on the patch, the build process, and how a tool like FlintRock makes deploying custom Spark clusters incredibly simple."

"Step 1. The Patch for Scala 2.13 Incompatibility The spark-sql-perf toolkit won't compile "out-of-the-box" with Spark 4.0+ due to its Scala 2.13 requirement. I've created a patch that resolves these compilation errors, which you can apply directly to the spark-sql-perf repository before building. The patch primarily involves updating Spark library dependencies in build.sbt and fixing code to be compatible with Spark 4.0+."

A patched version of spark-sql-perf is required to compile and run on Spark 4.0+ because of Scala 2.13 incompatibilities. The patch updates Spark library dependencies in build.sbt and adjusts code to restore compatibility. Build process: clone the spark-sql-perf repository, apply the patch, and assemble the benchmark JAR. Then build a custom Spark distribution and deploy it. Two usage options exist: include the patched assembly JAR in Spark's jars/ directory during the Spark build (which may require build script tweaks), or use the JAR at runtime on a deployed cluster. FlintRock simplifies provisioning custom Spark clusters on AWS.

Read at Medium