Apache Spark: Only the simple answer

Overview

In this post, I’m just gonna discuss some fundational things I learned about big data with Apache Spark. Personally, I’m just a bit interested in this topic, and do not aim to really become a big data professional (not yet~). It does take tremendous effort to learn Spark well, not to mention the entire big data ecosystem. I’ll update this post if I try out some new projects that really apply Spark and its APIs in a deep manner, but for now, let’s just talk about some basics of Spark.

Apache Spark vs Hadoop MapReduce

• MapReduce is a programming model, Spark is a processing framework

Important Components of Spark Ecosystem

• Core components[MOST IMPORTANT]:
  • Spark Core (Caching, RDD, DataFrames, Datasets || Transformations and Actions)
    1. Memory management
    2. Fault recovery
    3. Task dispatching
    4. Scheduling and monitoring jobs
  • Spark SQL (Data Query)
    • Used for structured and semi-structured data processing
    • Usually used for in-memory processing
    • Top level: DataFrame DSL(domain specific language) ; Spark SQl and HQL(Hive)
    • Level 2: DataFrame API
    • Level 3: Data Source API
    • Base Level: CSV + JSON + JDBC(Java Database connectivity) + etc storage/query
  • Spark Streaming (Stream data)
  • Spark MLlib (Machine Learnig toolkits)
  • GraphX (Graph Processing models)
• Langauge Support
  • Java
  • Python
  • Scala
  • R
• Spark Cluster Managers
  • Standalone mode: Default choice, run in FIFO order, and each application will try to use all available nodes
  • Apache YARN (Hadoop Integration): This is the resource manager of Hadoop, use this will help spark to connect to hdfs better
  • Kubernetes: For deployment, scaling and management of containerized applications

RDD

• Resilient Distributed Datasets
• RDDs are immutable, fault-tolerant distributed collections of objects that can be operated in parallel (split into partition and executed on different nodes of a cluster)
• 2 major types of operations:
  • Transformation: map, filter, join, union, etc. Yield a new RDD containing the result
  • Action: reduce, count, first, etc. Return a value after running a computation on RDD
• Works in a similar style as java Stream

How Spark runs applications with the help of its architecture

START EXECUTION

• In driver program
  • Spark applications runs as independent processes (i.e. split tasks) running across different machines
  • Spark sessions/context as the entry point of the application
  • Driver: Record the flow of the application
• Resource manager/Cluster manager (DAG Scheduler at the backend)
  • The driver program request resources from the clusters
  • Assign task to workers, one task per partition
  • Knows each step of the application for in-memory processing
• Worker node
  • processing slave (node manager) grands the request to usage of resources from resource manager
  • The request is called Container, within the Container, executor process is launched to apply tasks to its unit of work to the dataset in its partition and outputs a new partition dataset
  • because iterative algorithms apply operations repeated to the data, the benefit from caching datasets across iterations

END EXECUTION

• Results are sent back: worker node -> container -> manager -> driver program/disk
• The entire execution is lazily evaluated (transformations not evaluated until an action is called)

What is a Parquet file and what are its advantages

• Parquet is a columnar format that is supported byh several data processing systems (default data type for spark)
• Advisable to use if not all fields/columns in the data are used
• Advantages
  • able to fetch specific columns for access
  • consumes less space
  • follow type-specific encoding
  • limited I/O operations

What is shuffling in Spark? When does it occur?

• Shuffling is the process of redistributing data across partitions that may lead to data movement across executors
• Occurs while joining two tables or while performing byKey operations such as GroupByKey or ReduceByKey

Notes on Big Data Learning Journey (for those who truly want a Big Data job and for my future )

To excel in the Big Data domain, you should master the following skills:

1. Java & Scala $\implies$ Understand source code for related package/API development
2. Linux $\implies$ Everyone should know about shell scripts, bash and linux commands
3. Hadoop $\implies$ It’s a broad topic, but first of all, the ability to read whatever source code for an API is a must
4. Hive $\implies$ Know how to use it, understand how the SQL is converted in base code and how to optimize the query process or MapReduce/Spark Operations
5. Spark $\implies$ The core developement process (But honestly, most of the time it is still SQL)
6. Kafka $\implies$ High-volumn stream data processing; Good to use when you have high concurrency
7. Flink $\implies$ Faster than Spark sometimes. However, you should not discard Spark. Learn based on what you need.
8. HBase $\implies$ Know your database knowledge. Understand its fundamental knowledge
9. Zookeeper $\implies$ Distributation cluster data coordination services; Know how to use, better to understand the basic
10. YARN $\implies$ Cluster resources management; Know how to use
11. Sqoop, Flume, Oozie/Azkaban $\implies$ Know how to use

Different cluster managers

1. Spark Standalone mode
   • by default, applications submitted to the standalone mode cluster will run in FIFO order, and each application will try to use all available nodes
2. Apache Mesos
   • an open sources project to manage computer clusters, and can also run Hadoop applications
3. YARN
   • Apache YARN is the cluster resource manager of Hadoop 2.
4. Kubernetes
   • an open-source system for automating deployment, scaling and management of containerized applications