Data Engineer Interview Questions and Answers (4 Years Experience)
Briefly

Data Engineer Interview Questions and Answers (4 Years Experience)
"1. Scala Coding's Fibonacci Check Question: Write a Scala program to find out if a given number belongs to the Fibonacci series. If the response is yes, return that number. If not, return -1. Solution: For n = 7 → Output: -1 (7 is not in the Fibonacci series) For n = 8 → Output: 8 (8 is in the Fibonacci series)"
"2. What is YARN? Hadoop's resource management layer is called YARN (Yet Another Resource Negotiator). It controls how CPU and memory resources are distributed throughout the cluster. YARN increases the system's scalability and efficiency by separating resource management from job execution. Key components: Resource Manager (RM): Allocates cluster-wide resources. Node Manager (NM): Manages resources on each node. Application Master (AM): Manages execution of a single job."
The process consisted of two phases: an online assessment and a technical interview. The online assessment combined multiple-choice questions on Java, NoSQL, Hadoop, Spark, and Scala with two Scala programming problems to evaluate Scala fundamentals and problem-solving. The technical interview included coding and conceptual questions, for example a Scala program to check Fibonacci membership with outputs showing 7 → -1 and 8 → 8. The technical topics covered included YARN and its components (Resource Manager, Node Manager, Application Master) and Spark dynamic allocation. Dynamic allocation enables automatic scaling of executors to remove idle ones and add executors when tasks are pending. Important dynamic allocation parameters included spark.dynamicAllocation.enabled, spark.dynamicAllocation.minExecutors, and spark.dynamicAllocation.maxExecutors.
Read at Medium
Unable to calculate read time
[
|
]